Oct 30 00:42:08.030062 ip-10-0-140-206 systemd[1]: Starting Kubernetes Kubelet... Oct 30 00:42:08.396535 ip-10-0-140-206 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:42:08.396535 ip-10-0-140-206 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 00:42:08.396535 ip-10-0-140-206 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:42:08.396535 ip-10-0-140-206 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:42:08.400063 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.399985 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:42:08.400063 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400063 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:42:08.400063 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400066 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:42:08.400157 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400071 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:42:08.400157 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400074 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:42:08.400157 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400077 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:42:08.400157 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400080 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:42:08.400157 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400083 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:42:08.400157 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400086 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:42:08.400157 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400089 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:42:08.400157 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400092 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:42:08.400157 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400094 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:42:08.400157 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400097 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:42:08.400157 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400100 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:42:08.400157 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400103 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:42:08.400157 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400106 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:42:08.400157 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400109 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:42:08.400157 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400111 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:42:08.400157 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400121 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:42:08.400157 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400124 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:42:08.400157 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400129 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:42:08.400157 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400133 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:42:08.400592 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400136 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:42:08.400592 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400139 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:42:08.400592 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400142 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:42:08.400592 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400145 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:42:08.400592 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400147 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:42:08.400592 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400150 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:42:08.400592 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400153 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:42:08.400592 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400156 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:42:08.400592 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400159 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:42:08.400592 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400161 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:42:08.400592 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400164 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:42:08.400592 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400167 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:42:08.400592 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400170 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:42:08.400592 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400173 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:42:08.400592 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400175 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:42:08.400592 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400178 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:42:08.400592 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400181 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:42:08.400592 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400184 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:42:08.400592 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400186 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:42:08.400592 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400189 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:42:08.401099 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400192 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:42:08.401099 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400195 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:42:08.401099 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400198 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:42:08.401099 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400201 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:42:08.401099 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400204 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:42:08.401099 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400206 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:42:08.401099 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400209 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:42:08.401099 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400211 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:42:08.401099 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400214 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:42:08.401099 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400217 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:42:08.401099 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400220 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:42:08.401099 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400222 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:42:08.401099 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400224 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:42:08.401099 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400228 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:42:08.401099 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400231 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:42:08.401099 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400235 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:42:08.401099 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400237 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:42:08.401099 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400240 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:42:08.401099 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400243 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:42:08.401546 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400245 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:42:08.401546 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400248 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:42:08.401546 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400251 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:42:08.401546 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400253 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:42:08.401546 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400256 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:42:08.401546 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400259 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:42:08.401546 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400261 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:42:08.401546 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400264 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:42:08.401546 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400267 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:42:08.401546 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400270 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:42:08.401546 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400272 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:42:08.401546 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400275 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:42:08.401546 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400277 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:42:08.401546 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400280 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:42:08.401546 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400282 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:42:08.401546 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400285 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:42:08.401546 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400288 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:42:08.401546 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400290 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:42:08.401546 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400293 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:42:08.401546 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400295 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:42:08.402039 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400299 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:42:08.402039 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400303 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:42:08.402039 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400307 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:42:08.402039 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400310 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:42:08.402039 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400312 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:42:08.402039 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400315 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:42:08.402039 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400319 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:42:08.402039 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400321 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:42:08.402039 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400324 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:42:08.402039 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400327 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:42:08.402039 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400330 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:42:08.402039 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400333 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:42:08.402039 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400335 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:42:08.402039 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400338 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:42:08.402039 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400341 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:42:08.402039 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400343 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:42:08.402039 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400346 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:42:08.402039 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400348 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:42:08.402039 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400351 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:42:08.402488 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400354 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:42:08.402488 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400356 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:42:08.402488 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400359 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:42:08.402488 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400361 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:42:08.402488 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400364 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:42:08.402488 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.400367 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:42:08.402488 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401193 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:42:08.402488 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401202 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:42:08.402488 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401205 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:42:08.402488 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401209 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:42:08.402488 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401212 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:42:08.402488 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401215 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:42:08.402488 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401219 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:42:08.402488 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401222 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:42:08.402488 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401226 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:42:08.402488 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401229 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:42:08.402488 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401231 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:42:08.402488 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401234 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:42:08.402488 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401237 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:42:08.403006 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401240 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:42:08.403006 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401243 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:42:08.403006 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401246 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:42:08.403006 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401249 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:42:08.403006 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401251 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:42:08.403006 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401254 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:42:08.403006 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401257 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:42:08.403006 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401259 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:42:08.403006 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401262 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:42:08.403006 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401265 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:42:08.403006 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401267 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:42:08.403006 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401271 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:42:08.403006 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401274 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:42:08.403006 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401276 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:42:08.403006 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401279 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:42:08.403006 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401282 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:42:08.403006 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401284 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:42:08.403006 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401287 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:42:08.403006 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401290 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:42:08.403453 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401292 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:42:08.403453 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401295 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:42:08.403453 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401297 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:42:08.403453 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401300 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:42:08.403453 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401303 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:42:08.403453 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401305 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:42:08.403453 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401308 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:42:08.403453 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401310 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:42:08.403453 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401314 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:42:08.403453 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401317 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:42:08.403453 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401320 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:42:08.403453 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401323 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:42:08.403453 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401327 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:42:08.403453 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401331 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:42:08.403453 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401334 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:42:08.403453 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401337 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:42:08.403453 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401340 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:42:08.403453 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401342 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:42:08.403453 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401345 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:42:08.403453 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401348 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:42:08.403942 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401350 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:42:08.403942 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401353 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:42:08.403942 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401356 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:42:08.403942 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401358 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:42:08.403942 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401361 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:42:08.403942 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401363 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:42:08.403942 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401366 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:42:08.403942 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401368 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:42:08.403942 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401371 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:42:08.403942 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401373 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:42:08.403942 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401376 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:42:08.403942 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401378 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:42:08.403942 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401381 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:42:08.403942 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401384 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:42:08.403942 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401386 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:42:08.403942 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401389 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:42:08.403942 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401391 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:42:08.403942 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401394 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:42:08.403942 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401396 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:42:08.403942 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401399 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:42:08.404398 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401402 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:42:08.404398 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401405 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:42:08.404398 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401408 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:42:08.404398 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401411 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:42:08.404398 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401414 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:42:08.404398 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401417 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:42:08.404398 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401420 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:42:08.404398 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401424 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:42:08.404398 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401428 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:42:08.404398 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401431 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:42:08.404398 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401434 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:42:08.404398 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401437 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:42:08.404398 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401439 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:42:08.404398 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401442 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:42:08.404398 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401445 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:42:08.404398 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401448 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:42:08.404398 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401450 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:42:08.404398 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401453 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:42:08.404398 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401456 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:42:08.404863 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401458 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:42:08.404863 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401461 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:42:08.404863 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401463 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:42:08.404863 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401466 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:42:08.404863 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401469 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:42:08.404863 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401471 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:42:08.404863 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401474 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:42:08.404863 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401476 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:42:08.404863 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401479 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:42:08.404863 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401482 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:42:08.404863 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401486 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:42:08.404863 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401488 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:42:08.404863 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401491 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:42:08.404863 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401493 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:42:08.404863 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401496 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:42:08.404863 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.401997 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:42:08.404863 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402003 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:42:08.404863 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402006 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:42:08.404863 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402009 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:42:08.404863 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402012 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:42:08.405343 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402015 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:42:08.405343 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402018 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:42:08.405343 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402020 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:42:08.405343 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402023 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:42:08.405343 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402026 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:42:08.405343 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402028 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:42:08.405343 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402031 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:42:08.405343 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402034 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:42:08.405343 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402036 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:42:08.405343 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402039 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:42:08.405343 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402042 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:42:08.405343 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402045 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:42:08.405343 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402048 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:42:08.405343 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402050 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:42:08.405343 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402053 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:42:08.405343 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402056 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:42:08.405343 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402059 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:42:08.405343 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402062 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:42:08.405343 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402065 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:42:08.405343 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402068 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:42:08.405835 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402071 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:42:08.405835 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402073 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:42:08.405835 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402076 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:42:08.405835 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402078 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:42:08.405835 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402081 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:42:08.405835 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402083 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:42:08.405835 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402086 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:42:08.405835 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402089 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:42:08.405835 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402092 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:42:08.405835 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402095 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:42:08.405835 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402098 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:42:08.405835 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402101 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:42:08.405835 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402103 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:42:08.405835 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402106 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:42:08.405835 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402108 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:42:08.405835 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402111 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:42:08.405835 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402114 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:42:08.405835 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402116 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:42:08.405835 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402119 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:42:08.406294 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402123 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:42:08.406294 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402126 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:42:08.406294 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402130 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:42:08.406294 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402133 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:42:08.406294 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402136 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:42:08.406294 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402138 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:42:08.406294 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402141 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:42:08.406294 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402144 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:42:08.406294 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402146 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:42:08.406294 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402149 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:42:08.406294 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402151 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:42:08.406294 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402154 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:42:08.406294 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402157 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:42:08.406294 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402160 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:42:08.406294 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402163 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:42:08.406294 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402166 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:42:08.406294 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402168 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:42:08.406294 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402171 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:42:08.406294 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402174 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:42:08.406738 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402176 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:42:08.406738 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402179 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:42:08.406738 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402181 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:42:08.406738 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402185 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:42:08.406738 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402188 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:42:08.406738 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402191 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:42:08.406738 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402194 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:42:08.406738 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402196 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:42:08.406738 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402199 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:42:08.406738 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402202 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:42:08.406738 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402204 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:42:08.406738 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402207 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:42:08.406738 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402210 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:42:08.406738 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402212 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:42:08.406738 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402215 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:42:08.406738 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402218 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:42:08.406738 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402220 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:42:08.406738 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402223 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:42:08.406738 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402225 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:42:08.406738 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402228 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:42:08.407283 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402231 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:42:08.407283 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402234 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:42:08.407283 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402236 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:42:08.407283 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402239 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:42:08.407283 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402246 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:42:08.407283 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402249 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:42:08.407283 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402252 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:42:08.407283 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402255 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:42:08.407283 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402258 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:42:08.407283 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402261 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:42:08.407283 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402263 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:42:08.407283 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402266 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:42:08.407283 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402268 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:42:08.407283 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402271 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:42:08.407283 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402273 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:42:08.407283 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402276 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:42:08.407283 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402281 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:42:08.407283 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402284 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:42:08.407283 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402286 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:42:08.407720 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402289 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:42:08.407720 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402291 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:42:08.407720 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402294 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:42:08.407720 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.402296 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:42:08.407720 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403459 2573 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 00:42:08.407720 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403470 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 00:42:08.407720 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403477 2573 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 00:42:08.407720 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403483 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 00:42:08.407720 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403487 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 00:42:08.407720 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403490 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 00:42:08.407720 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403495 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 00:42:08.407720 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403499 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 00:42:08.407720 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403503 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 00:42:08.407720 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403506 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 00:42:08.407720 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403510 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 00:42:08.407720 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403513 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 00:42:08.407720 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403517 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 00:42:08.407720 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403520 2573 flags.go:64] FLAG: --cgroup-root="" Oct 30 00:42:08.407720 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403523 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 00:42:08.407720 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403526 2573 flags.go:64] FLAG: --client-ca-file="" Oct 30 00:42:08.407720 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403529 2573 flags.go:64] FLAG: --cloud-provider="external" Oct 30 00:42:08.407720 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403533 2573 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 00:42:08.407720 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403537 2573 flags.go:64] FLAG: --cluster-domain="" Oct 30 00:42:08.407720 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403540 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 00:42:08.408291 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403544 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 00:42:08.408291 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403547 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 00:42:08.408291 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403551 2573 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 00:42:08.408291 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403555 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 00:42:08.408291 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403558 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 00:42:08.408291 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403561 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 00:42:08.408291 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403565 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 00:42:08.408291 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403568 2573 flags.go:64] FLAG: --contention-profiling="false" Oct 30 00:42:08.408291 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403571 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 00:42:08.408291 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403574 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 00:42:08.408291 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403578 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 00:42:08.408291 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403581 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 00:42:08.408291 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403585 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 00:42:08.408291 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403588 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 00:42:08.408291 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403591 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 00:42:08.408291 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403594 2573 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 00:42:08.408291 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403597 2573 flags.go:64] FLAG: --enable-server="true" Oct 30 00:42:08.408291 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403601 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 00:42:08.408291 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403606 2573 flags.go:64] FLAG: --event-burst="100" Oct 30 00:42:08.408291 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403609 2573 flags.go:64] FLAG: --event-qps="50" Oct 30 00:42:08.408291 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403613 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 00:42:08.408291 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403616 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 00:42:08.408291 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403619 2573 flags.go:64] FLAG: --eviction-hard="" Oct 30 00:42:08.408291 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403623 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 00:42:08.408889 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403626 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 00:42:08.408889 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403630 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 00:42:08.408889 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403633 2573 flags.go:64] FLAG: --eviction-soft="" Oct 30 00:42:08.408889 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403636 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 00:42:08.408889 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403639 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 00:42:08.408889 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403642 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 00:42:08.408889 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403645 2573 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 00:42:08.408889 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403649 2573 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 00:42:08.408889 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403652 2573 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 00:42:08.408889 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403655 2573 flags.go:64] FLAG: --feature-gates="" Oct 30 00:42:08.408889 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403659 2573 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 00:42:08.408889 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403663 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 00:42:08.408889 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403666 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 00:42:08.408889 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403670 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 00:42:08.408889 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403673 2573 flags.go:64] FLAG: --healthz-port="10248" Oct 30 00:42:08.408889 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403676 2573 flags.go:64] FLAG: --help="false" Oct 30 00:42:08.408889 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403679 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-140-206.ec2.internal" Oct 30 00:42:08.408889 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403683 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 00:42:08.408889 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403686 2573 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 00:42:08.408889 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403689 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 00:42:08.408889 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403692 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 00:42:08.408889 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403695 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 00:42:08.408889 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403699 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 00:42:08.408889 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403701 2573 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 00:42:08.409459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403704 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 00:42:08.409459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403707 2573 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 00:42:08.409459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403711 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 00:42:08.409459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403714 2573 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 00:42:08.409459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403717 2573 flags.go:64] FLAG: --kube-reserved="" Oct 30 00:42:08.409459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403720 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 00:42:08.409459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403723 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 00:42:08.409459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403726 2573 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 00:42:08.409459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403729 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 00:42:08.409459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403732 2573 flags.go:64] FLAG: --lock-file="" Oct 30 00:42:08.409459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403735 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 00:42:08.409459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403738 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 00:42:08.409459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403741 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 00:42:08.409459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403747 2573 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 00:42:08.409459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403750 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 00:42:08.409459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403753 2573 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 00:42:08.409459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403756 2573 flags.go:64] FLAG: --logging-format="text" Oct 30 00:42:08.409459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403769 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 00:42:08.409459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403773 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 00:42:08.409459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403776 2573 flags.go:64] FLAG: --manifest-url="" Oct 30 00:42:08.409459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403779 2573 flags.go:64] FLAG: --manifest-url-header="" Oct 30 00:42:08.409459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403784 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 00:42:08.409459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403787 2573 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 00:42:08.409459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403791 2573 flags.go:64] FLAG: --max-pods="110" Oct 30 00:42:08.409459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403794 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 00:42:08.410111 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403798 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 00:42:08.410111 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403801 2573 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 00:42:08.410111 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403804 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 00:42:08.410111 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403807 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 00:42:08.410111 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403810 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 00:42:08.410111 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403814 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 00:42:08.410111 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403821 2573 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 00:42:08.410111 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403825 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 00:42:08.410111 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403828 2573 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 00:42:08.410111 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403831 2573 flags.go:64] FLAG: --pod-cidr="" Oct 30 00:42:08.410111 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403834 2573 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 00:42:08.410111 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403838 2573 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 00:42:08.410111 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403841 2573 flags.go:64] FLAG: --pods-per-core="0" Oct 30 00:42:08.410111 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403844 2573 flags.go:64] FLAG: --port="10250" Oct 30 00:42:08.410111 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403847 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 00:42:08.410111 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403850 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e015b2d1ed84cca5" Oct 30 00:42:08.410111 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403854 2573 flags.go:64] FLAG: --qos-reserved="" Oct 30 00:42:08.410111 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403857 2573 flags.go:64] FLAG: --read-only-port="10255" Oct 30 00:42:08.410111 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403860 2573 flags.go:64] FLAG: --register-node="true" Oct 30 00:42:08.410111 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403863 2573 flags.go:64] FLAG: --register-with-taints="" Oct 30 00:42:08.410111 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403867 2573 flags.go:64] FLAG: --registry-burst="10" Oct 30 00:42:08.410111 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403870 2573 flags.go:64] FLAG: --registry-qps="5" Oct 30 00:42:08.410111 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403873 2573 flags.go:64] FLAG: --reserved-cpus="" Oct 30 00:42:08.410111 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403876 2573 flags.go:64] FLAG: --reserved-memory="" Oct 30 00:42:08.410111 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403879 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 00:42:08.410111 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403883 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 00:42:08.410746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403886 2573 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 00:42:08.410746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403891 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 00:42:08.410746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403894 2573 flags.go:64] FLAG: --runonce="false" Oct 30 00:42:08.410746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403898 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 00:42:08.410746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403901 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 00:42:08.410746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403904 2573 flags.go:64] FLAG: --seccomp-default="false" Oct 30 00:42:08.410746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403907 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 00:42:08.410746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403911 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 00:42:08.410746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403914 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 00:42:08.410746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403917 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 00:42:08.410746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403920 2573 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 00:42:08.410746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403923 2573 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 00:42:08.410746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403926 2573 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 00:42:08.410746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403929 2573 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 00:42:08.410746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403932 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 00:42:08.410746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403936 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 00:42:08.410746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403939 2573 flags.go:64] FLAG: --system-cgroups="" Oct 30 00:42:08.410746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403942 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 00:42:08.410746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403947 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 00:42:08.410746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403951 2573 flags.go:64] FLAG: --tls-cert-file="" Oct 30 00:42:08.410746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403954 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 00:42:08.410746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403958 2573 flags.go:64] FLAG: --tls-min-version="" Oct 30 00:42:08.410746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403961 2573 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 00:42:08.410746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403964 2573 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 00:42:08.410746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403968 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 00:42:08.411433 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403971 2573 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 00:42:08.411433 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403974 2573 flags.go:64] FLAG: --v="2" Oct 30 00:42:08.411433 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403979 2573 flags.go:64] FLAG: --version="false" Oct 30 00:42:08.411433 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403983 2573 flags.go:64] FLAG: --vmodule="" Oct 30 00:42:08.411433 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403987 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 00:42:08.411433 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.403991 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 00:42:08.411433 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404128 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:42:08.411433 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404132 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:42:08.411433 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404136 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:42:08.411433 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404138 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:42:08.411433 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404141 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:42:08.411433 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404144 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:42:08.411433 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404147 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:42:08.411433 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404150 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:42:08.411433 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404153 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:42:08.411433 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404155 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:42:08.411433 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404158 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:42:08.411433 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404161 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:42:08.411433 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404163 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:42:08.411433 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404166 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:42:08.411938 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404170 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:42:08.411938 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404174 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:42:08.411938 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404177 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:42:08.411938 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404180 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:42:08.411938 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404185 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:42:08.411938 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404189 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:42:08.411938 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404192 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:42:08.411938 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404195 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:42:08.411938 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404198 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:42:08.411938 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404200 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:42:08.411938 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404203 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:42:08.411938 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404206 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:42:08.411938 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404209 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:42:08.411938 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404212 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:42:08.411938 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404214 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:42:08.411938 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404217 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:42:08.411938 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404220 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:42:08.411938 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404222 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:42:08.411938 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404225 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:42:08.412382 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404228 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:42:08.412382 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404231 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:42:08.412382 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404234 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:42:08.412382 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404237 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:42:08.412382 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404239 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:42:08.412382 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404243 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:42:08.412382 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404247 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:42:08.412382 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404251 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:42:08.412382 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404255 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:42:08.412382 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404258 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:42:08.412382 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404261 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:42:08.412382 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404264 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:42:08.412382 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404266 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:42:08.412382 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404269 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:42:08.412382 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404272 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:42:08.412382 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404274 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:42:08.412382 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404277 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:42:08.412382 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404281 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:42:08.412382 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404283 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:42:08.412382 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404286 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:42:08.412880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404289 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:42:08.412880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404291 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:42:08.412880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404294 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:42:08.412880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404297 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:42:08.412880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404300 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:42:08.412880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404302 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:42:08.412880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404305 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:42:08.412880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404308 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:42:08.412880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404310 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:42:08.412880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404313 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:42:08.412880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404315 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:42:08.412880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404318 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:42:08.412880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404321 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:42:08.412880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404325 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:42:08.412880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404327 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:42:08.412880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404330 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:42:08.412880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404333 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:42:08.412880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404336 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:42:08.412880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404339 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:42:08.413331 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404342 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:42:08.413331 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404344 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:42:08.413331 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404347 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:42:08.413331 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404350 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:42:08.413331 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404353 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:42:08.413331 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404355 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:42:08.413331 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404358 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:42:08.413331 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404360 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:42:08.413331 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404363 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:42:08.413331 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404366 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:42:08.413331 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404370 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:42:08.413331 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404372 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:42:08.413331 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404375 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:42:08.413331 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404378 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:42:08.413331 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404380 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:42:08.413331 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404383 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:42:08.413331 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404386 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:42:08.413331 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404388 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:42:08.413331 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404391 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:42:08.413331 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404394 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:42:08.413895 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404397 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:42:08.413895 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404399 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:42:08.413895 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404402 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:42:08.413895 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404405 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:42:08.413895 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404408 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:42:08.413895 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404411 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:42:08.413895 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404415 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:42:08.413895 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404418 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:42:08.413895 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404421 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:42:08.413895 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404423 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:42:08.413895 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404426 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:42:08.413895 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404428 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:42:08.413895 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404431 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:42:08.413895 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404434 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:42:08.413895 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.404459 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:42:08.413895 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404500 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:42:08.414270 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404504 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:42:08.414270 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404507 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:42:08.414270 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404510 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:42:08.414270 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404513 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:42:08.414270 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404516 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:42:08.414270 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404518 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:42:08.414270 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404523 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:42:08.414270 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404526 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:42:08.414270 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404528 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:42:08.414270 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404532 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:42:08.414270 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404536 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:42:08.414270 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404539 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:42:08.414270 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404542 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:42:08.414270 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404545 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:42:08.414270 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404548 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:42:08.414270 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404551 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:42:08.414270 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404554 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:42:08.414270 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404556 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:42:08.414270 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404559 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:42:08.414716 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404561 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:42:08.414716 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404564 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:42:08.414716 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404567 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:42:08.414716 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404569 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:42:08.414716 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404573 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:42:08.414716 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404576 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:42:08.414716 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404578 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:42:08.414716 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404581 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:42:08.414716 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404584 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:42:08.414716 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404586 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:42:08.414716 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404589 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:42:08.414716 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404592 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:42:08.414716 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404595 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:42:08.414716 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404598 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:42:08.414716 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404602 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:42:08.414716 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404605 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:42:08.414716 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404608 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:42:08.414716 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404611 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:42:08.414716 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404614 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:42:08.414716 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404618 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:42:08.415205 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404621 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:42:08.415205 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404623 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:42:08.415205 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404626 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:42:08.415205 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404629 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:42:08.415205 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404632 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:42:08.415205 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404634 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:42:08.415205 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404637 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:42:08.415205 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404639 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:42:08.415205 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404642 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:42:08.415205 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404645 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:42:08.415205 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404647 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:42:08.415205 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404651 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:42:08.415205 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404653 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:42:08.415205 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404656 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:42:08.415205 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404658 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:42:08.415205 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404661 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:42:08.415205 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404665 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:42:08.415205 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404668 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:42:08.415205 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404671 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:42:08.415205 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404673 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:42:08.415661 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404676 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:42:08.415661 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404679 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:42:08.415661 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404682 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:42:08.415661 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404685 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:42:08.415661 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404687 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:42:08.415661 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404690 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:42:08.415661 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404693 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:42:08.415661 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404696 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:42:08.415661 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404699 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:42:08.415661 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404702 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:42:08.415661 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404704 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:42:08.415661 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404708 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:42:08.415661 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404711 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:42:08.415661 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404714 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:42:08.415661 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404716 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:42:08.415661 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404719 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:42:08.415661 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404722 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:42:08.415661 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404724 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:42:08.415661 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404727 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:42:08.416153 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404729 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:42:08.416153 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404732 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:42:08.416153 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404735 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:42:08.416153 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404737 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:42:08.416153 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404740 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:42:08.416153 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404743 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:42:08.416153 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404746 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:42:08.416153 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404748 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:42:08.416153 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404751 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:42:08.416153 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404755 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:42:08.416153 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404768 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:42:08.416153 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404771 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:42:08.416153 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404774 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:42:08.416153 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404777 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:42:08.416153 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404780 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:42:08.416153 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404782 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:42:08.416153 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404785 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:42:08.416153 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404788 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:42:08.416153 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404791 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:42:08.416153 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404794 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:42:08.416611 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404797 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:42:08.416611 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404800 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:42:08.416611 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404803 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:42:08.416611 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404806 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:42:08.416611 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404810 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:42:08.416611 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404813 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:42:08.416611 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.404816 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:42:08.416611 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.411956 2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 00:42:08.416611 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.411970 2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 00:42:08.416611 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.411987 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 00:42:08.416611 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.411991 2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 00:42:08.416611 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412047 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:42:08.416611 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412053 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:42:08.416611 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412057 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:42:08.416611 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412060 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:42:08.416611 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412063 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:42:08.416611 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412065 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:42:08.416611 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412068 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:42:08.416611 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412071 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:42:08.417065 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412074 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:42:08.417065 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412077 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:42:08.417065 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412079 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:42:08.417065 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412082 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:42:08.417065 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412085 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:42:08.417065 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412087 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:42:08.417065 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412090 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:42:08.417065 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412093 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:42:08.417065 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412096 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:42:08.417065 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412098 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:42:08.417065 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412101 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:42:08.417065 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412104 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:42:08.417065 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412106 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:42:08.417065 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412110 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:42:08.417065 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412113 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:42:08.417065 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412116 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:42:08.417065 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412118 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:42:08.417065 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412121 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:42:08.417065 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412124 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:42:08.417515 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412127 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:42:08.417515 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412130 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:42:08.417515 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412133 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:42:08.417515 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412136 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:42:08.417515 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412138 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:42:08.417515 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412141 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:42:08.417515 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412144 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:42:08.417515 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412146 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:42:08.417515 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412150 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:42:08.417515 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412152 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:42:08.417515 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412155 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:42:08.417515 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412158 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:42:08.417515 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412160 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:42:08.417515 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412163 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:42:08.417515 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412166 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:42:08.417515 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412169 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:42:08.417515 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412171 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:42:08.417515 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412174 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:42:08.417515 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412176 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:42:08.417515 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412179 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:42:08.418028 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412182 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:42:08.418028 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412185 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:42:08.418028 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412187 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:42:08.418028 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412190 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:42:08.418028 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412192 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:42:08.418028 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412195 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:42:08.418028 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412198 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:42:08.418028 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412201 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:42:08.418028 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412204 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:42:08.418028 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412206 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:42:08.418028 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412209 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:42:08.418028 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412212 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:42:08.418028 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412216 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:42:08.418028 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412220 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:42:08.418028 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412224 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:42:08.418028 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412226 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:42:08.418028 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412229 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:42:08.418028 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412232 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:42:08.418028 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412235 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:42:08.418028 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412237 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:42:08.418489 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412240 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:42:08.418489 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412242 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:42:08.418489 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412245 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:42:08.418489 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412248 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:42:08.418489 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412251 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:42:08.418489 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412253 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:42:08.418489 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412256 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:42:08.418489 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412259 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:42:08.418489 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412262 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:42:08.418489 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412264 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:42:08.418489 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412267 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:42:08.418489 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412269 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:42:08.418489 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412272 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:42:08.418489 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412274 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:42:08.418489 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412277 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:42:08.418489 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412280 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:42:08.418489 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412283 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:42:08.418489 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412285 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:42:08.418489 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412288 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:42:08.418956 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412291 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:42:08.418956 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412294 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:42:08.418956 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412297 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:42:08.418956 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412300 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:42:08.418956 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412303 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:42:08.418956 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412306 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:42:08.418956 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412308 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:42:08.418956 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412311 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:42:08.418956 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412314 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:42:08.418956 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412316 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:42:08.418956 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412319 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:42:08.418956 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412322 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:42:08.418956 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412324 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:42:08.418956 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412328 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:42:08.418956 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412332 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:42:08.418956 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412335 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:42:08.418956 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412338 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:42:08.418956 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412341 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:42:08.418956 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412343 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:42:08.419402 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412346 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:42:08.419402 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.412373 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:42:08.419402 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412510 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:42:08.419402 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412515 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:42:08.419402 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412518 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:42:08.419402 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412521 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:42:08.419402 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412524 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:42:08.419402 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412526 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:42:08.419402 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412529 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:42:08.419402 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412532 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:42:08.419402 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412535 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:42:08.419402 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412537 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:42:08.419402 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412540 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:42:08.419402 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412542 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:42:08.419402 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412545 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:42:08.419775 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412549 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:42:08.419775 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412554 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:42:08.419775 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412557 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:42:08.419775 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412560 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:42:08.419775 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412563 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:42:08.419775 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412566 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:42:08.419775 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412569 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:42:08.419775 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412573 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:42:08.419775 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412575 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:42:08.419775 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412578 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:42:08.419775 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412580 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:42:08.419775 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412584 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:42:08.419775 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412588 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:42:08.419775 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412590 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:42:08.419775 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412593 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:42:08.419775 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412596 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:42:08.419775 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412598 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:42:08.419775 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412601 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:42:08.419775 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412604 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:42:08.420317 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412606 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:42:08.420317 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412609 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:42:08.420317 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412611 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:42:08.420317 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412614 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:42:08.420317 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412617 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:42:08.420317 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412619 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:42:08.420317 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412622 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:42:08.420317 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412624 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:42:08.420317 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412627 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:42:08.420317 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412630 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:42:08.420317 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412632 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:42:08.420317 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412635 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:42:08.420317 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412638 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:42:08.420317 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412640 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:42:08.420317 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412644 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:42:08.420317 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412646 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:42:08.420317 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412649 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:42:08.420317 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412651 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:42:08.420317 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412654 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:42:08.420852 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412657 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:42:08.420852 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412660 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:42:08.420852 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412662 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:42:08.420852 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412665 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:42:08.420852 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412668 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:42:08.420852 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412670 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:42:08.420852 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412673 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:42:08.420852 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412676 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:42:08.420852 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412678 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:42:08.420852 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412681 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:42:08.420852 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412683 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:42:08.420852 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412686 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:42:08.420852 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412689 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:42:08.420852 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412691 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:42:08.420852 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412694 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:42:08.420852 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412696 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:42:08.420852 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412699 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:42:08.420852 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412702 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:42:08.420852 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412704 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:42:08.420852 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412707 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:42:08.421553 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412709 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:42:08.421553 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412712 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:42:08.421553 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412714 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:42:08.421553 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412717 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:42:08.421553 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412719 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:42:08.421553 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412722 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:42:08.421553 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412725 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:42:08.421553 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412728 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:42:08.421553 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412731 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:42:08.421553 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412733 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:42:08.421553 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412736 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:42:08.421553 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412739 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:42:08.421553 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412741 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:42:08.421553 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412744 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:42:08.421553 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412746 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:42:08.421553 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412749 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:42:08.421553 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412751 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:42:08.421553 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412754 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:42:08.421553 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412756 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:42:08.421553 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412771 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:42:08.422040 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412774 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:42:08.422040 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412776 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:42:08.422040 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412779 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:42:08.422040 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412782 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:42:08.422040 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412785 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:42:08.422040 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412787 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:42:08.422040 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412790 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:42:08.422040 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412792 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:42:08.422040 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412795 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:42:08.422040 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412798 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:42:08.422040 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412800 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:42:08.422040 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412803 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:42:08.422040 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412806 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:42:08.422040 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412808 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:42:08.422040 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412811 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:42:08.422040 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.412835 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:42:08.422416 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412867 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:42:08.422416 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412871 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:42:08.422416 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412874 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:42:08.422416 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412877 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:42:08.422416 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412880 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:42:08.422416 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412884 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:42:08.422416 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412887 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:42:08.422416 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412890 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:42:08.422416 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412893 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:42:08.422416 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412896 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:42:08.422416 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412898 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:42:08.422416 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412901 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:42:08.422416 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412904 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:42:08.422416 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412908 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:42:08.422416 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412911 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:42:08.422416 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412913 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:42:08.422416 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412916 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:42:08.422416 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412918 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:42:08.422416 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412921 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:42:08.422880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412923 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:42:08.422880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412926 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:42:08.422880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412928 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:42:08.422880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412931 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:42:08.422880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412935 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:42:08.422880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412939 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:42:08.422880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412942 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:42:08.422880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412945 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:42:08.422880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412947 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:42:08.422880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412950 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:42:08.422880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412953 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:42:08.422880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412955 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:42:08.422880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412958 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:42:08.422880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412961 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:42:08.422880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412963 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:42:08.422880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412966 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:42:08.422880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412968 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:42:08.422880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412971 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:42:08.422880 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412973 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:42:08.423334 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412976 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:42:08.423334 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412980 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:42:08.423334 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412982 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:42:08.423334 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412985 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:42:08.423334 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412991 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:42:08.423334 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412994 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:42:08.423334 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412997 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:42:08.423334 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.412999 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:42:08.423334 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413002 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:42:08.423334 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413005 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:42:08.423334 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413008 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:42:08.423334 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413011 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:42:08.423334 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413013 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:42:08.423334 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413016 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:42:08.423334 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413019 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:42:08.423334 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413022 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:42:08.423334 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413025 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:42:08.423334 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413027 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:42:08.423334 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413030 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:42:08.423334 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413033 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:42:08.423871 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413036 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:42:08.423871 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413038 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:42:08.423871 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413041 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:42:08.423871 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413043 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:42:08.423871 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413046 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:42:08.423871 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413049 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:42:08.423871 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413052 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:42:08.423871 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413054 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:42:08.423871 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413057 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:42:08.423871 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413059 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:42:08.423871 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413062 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:42:08.423871 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413064 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:42:08.423871 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413067 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:42:08.423871 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413075 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:42:08.423871 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413077 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:42:08.423871 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413080 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:42:08.423871 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413084 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:42:08.423871 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413087 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:42:08.423871 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413090 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:42:08.424318 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413092 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:42:08.424318 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413095 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:42:08.424318 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413098 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:42:08.424318 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413100 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:42:08.424318 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413103 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:42:08.424318 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413105 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:42:08.424318 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413108 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:42:08.424318 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413110 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:42:08.424318 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413113 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:42:08.424318 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413116 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:42:08.424318 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413118 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:42:08.424318 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413121 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:42:08.424318 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413123 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:42:08.424318 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413126 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:42:08.424318 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413129 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:42:08.424318 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413132 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:42:08.424318 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413134 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:42:08.424318 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413137 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:42:08.424318 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413139 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:42:08.424318 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413142 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:42:08.424843 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413145 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:42:08.424843 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413147 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:42:08.424843 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413150 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:42:08.424843 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413153 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:42:08.424843 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413155 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:42:08.424843 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413158 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:42:08.424843 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413160 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:42:08.424843 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413163 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:42:08.424843 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:08.413167 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:42:08.424843 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.413217 2573 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 00:42:08.424843 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.415657 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 00:42:08.424843 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.416804 2573 server.go:1014] "Starting client certificate rotation" Oct 30 00:42:08.424843 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.416898 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 00:42:08.424843 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.417789 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 00:42:08.424843 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.420938 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:42:08.424843 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.423103 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:42:08.436211 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.436184 2573 log.go:25] "Validated CRI v1 runtime API" Oct 30 00:42:08.439495 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.439480 2573 log.go:25] "Validated CRI v1 image API" Oct 30 00:42:08.440260 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.440237 2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 00:42:08.446378 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.446355 2573 fs.go:135] Filesystem UUIDs: map[1b29b2ea-801b-49c1-b11e-b5b8de5e0a85:/dev/nvme0n1p4 70925af6-95b4-43ec-834e-1817dce62819:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 00:42:08.446448 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.446376 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 00:42:08.453161 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.453048 2573 manager.go:217] Machine: {Timestamp:2026-10-30 00:42:08.450486296 +0000 UTC m=+0.324217438 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096541 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20b867d17fe0b98927a8b0bdf4f4e8 SystemUUID:ec20b867-d17f-e0b9-8927-a8b0bdf4f4e8 BootID:e6864bfd-24c6-4313-986b-17c0c493cdb0 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e3:d0:ff:f3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e3:d0:ff:f3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:12:42:41:b9:6e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 00:42:08.453828 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.453817 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 00:42:08.453914 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.453903 2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 00:42:08.455185 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.455157 2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 00:42:08.455317 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.455186 2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-206.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 00:42:08.455358 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.455320 2573 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 00:42:08.455358 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.455328 2573 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 00:42:08.455423 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.455366 2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:42:08.455423 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.455384 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:42:08.456659 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.456649 2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 00:42:08.461295 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.461281 2573 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 00:42:08.461420 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.461411 2573 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 00:42:08.462237 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.462228 2573 kubelet.go:493] "Attempting to sync node with API server" Oct 30 00:42:08.462308 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.462243 2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 00:42:08.462308 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.462255 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 00:42:08.462308 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.462264 2573 kubelet.go:405] "Adding apiserver pod source" Oct 30 00:42:08.462308 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.462277 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 00:42:08.462613 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.462599 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:42:08.465195 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.465178 2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 00:42:08.473917 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.473901 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:42:08.474826 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.474813 2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 00:42:08.474877 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.474844 2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 00:42:08.475857 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.475835 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 00:42:08.475857 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.475862 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 00:42:08.476000 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.475869 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 00:42:08.476000 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.475875 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 00:42:08.476000 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.475881 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 00:42:08.476000 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.475887 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 00:42:08.476000 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.475896 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 00:42:08.476000 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.475903 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 00:42:08.476000 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.475911 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 00:42:08.476000 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.475916 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 00:42:08.476000 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.475925 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 00:42:08.476000 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.475934 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 00:42:08.477365 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.477349 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 00:42:08.477365 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.477365 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 00:42:08.481432 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.481418 2573 server.go:1263] "Started kubelet" Oct 30 00:42:08.481524 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.481497 2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 00:42:08.481821 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.481749 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 00:42:08.481900 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.481841 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 00:42:08.482268 ip-10-0-140-206 systemd[1]: Started Kubernetes Kubelet. Oct 30 00:42:08.482790 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.482744 2573 server.go:317] "Adding debug handlers to kubelet server" Oct 30 00:42:08.483411 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.483395 2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 00:42:08.488509 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:08.488488 2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 00:42:08.505806 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.505787 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 00:42:08.505889 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.505853 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j7s9g" Oct 30 00:42:08.505938 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.505885 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-206.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 00:42:08.507083 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:08.505882 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-206.ec2.internal.18e327e20e681951 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-206.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-206.ec2.internal,},FirstTimestamp:2026-10-30 00:42:08.481384785 +0000 UTC m=+0.355115927,LastTimestamp:2026-10-30 00:42:08.481384785 +0000 UTC m=+0.355115927,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-206.ec2.internal,}" Oct 30 00:42:08.510568 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.510550 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j7s9g" Oct 30 00:42:08.528029 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:08.528002 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 00:42:08.528029 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:08.528004 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-206.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 00:42:08.621882 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.621851 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 00:42:08.623222 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.623203 2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 00:42:08.623336 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:08.623319 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-206.ec2.internal\" not found" Oct 30 00:42:08.623983 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.623966 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 00:42:08.623983 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.623970 2573 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 00:42:08.624135 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.623992 2573 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 00:42:08.624135 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.624091 2573 reconstruct.go:97] "Volume reconstruction finished" Oct 30 00:42:08.624135 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.624100 2573 reconciler.go:29] "Reconciler: start to sync state" Oct 30 00:42:08.624221 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.624214 2573 factory.go:55] Registering systemd factory Oct 30 00:42:08.624255 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.624225 2573 factory.go:223] Registration of the systemd container factory successfully Oct 30 00:42:08.624439 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.624426 2573 factory.go:153] Registering CRI-O factory Oct 30 00:42:08.624501 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.624443 2573 factory.go:223] Registration of the crio container factory successfully Oct 30 00:42:08.624549 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.624504 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 00:42:08.624549 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.624523 2573 factory.go:103] Registering Raw factory Oct 30 00:42:08.624549 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.624538 2573 manager.go:1199] Started watching for new ooms in manager Oct 30 00:42:08.626138 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.626119 2573 manager.go:319] Starting recovery of all containers Oct 30 00:42:08.636575 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.636555 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:42:08.636710 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.636575 2573 manager.go:324] Recovery completed Oct 30 00:42:08.640540 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.640528 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:42:08.642350 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.642333 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-206.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:42:08.642417 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.642366 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-206.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:42:08.642417 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.642379 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-206.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:42:08.642916 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.642903 2573 cpu_manager.go:226] "Starting" policy="none" Oct 30 00:42:08.642916 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.642915 2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 00:42:08.643026 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.642930 2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 00:42:08.643026 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.642909 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:42:08.643622 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.643603 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-206.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:42:08.643697 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.643637 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-206.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:42:08.643697 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.643649 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-206.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:42:08.645168 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.645155 2573 policy_none.go:50] "Start" Oct 30 00:42:08.645228 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.645177 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 00:42:08.645228 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.645187 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 00:42:08.647223 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.647178 2573 policy_none.go:44] "Start" Oct 30 00:42:08.649042 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:08.649025 2573 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-206.ec2.internal\" not found" node="ip-10-0-140-206.ec2.internal" Oct 30 00:42:08.689439 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.689284 2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 00:42:08.693793 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.693777 2573 manager.go:341] "Starting Device Plugin manager" Oct 30 00:42:08.693866 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:08.693808 2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 00:42:08.693866 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.693817 2573 server.go:87] "Starting device plugin registration server" Oct 30 00:42:08.694057 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.694045 2573 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 00:42:08.694106 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.694057 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 00:42:08.694169 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.694149 2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 00:42:08.694230 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.694219 2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 00:42:08.694230 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.694230 2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 00:42:08.694804 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:08.694787 2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 00:42:08.694891 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:08.694835 2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-206.ec2.internal\" not found" Oct 30 00:42:08.744716 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.744676 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 00:42:08.746859 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.746073 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 00:42:08.746859 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.746091 2573 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 00:42:08.746859 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.746110 2573 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 00:42:08.746859 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:08.746140 2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 00:42:08.759386 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.759366 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:42:08.794187 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.794167 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:42:08.795806 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.795281 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-206.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:42:08.795806 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.795312 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-206.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:42:08.795806 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.795327 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-206.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:42:08.795806 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.795358 2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-206.ec2.internal" Oct 30 00:42:08.822110 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.822087 2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-206.ec2.internal" Oct 30 00:42:08.822110 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:08.822115 2573 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-206.ec2.internal\": node \"ip-10-0-140-206.ec2.internal\" not found" Oct 30 00:42:08.846527 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.846491 2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-206.ec2.internal"] Oct 30 00:42:08.846633 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.846571 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:42:08.848858 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.848838 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-206.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:42:08.848950 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.848873 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-206.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:42:08.848950 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.848883 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-206.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:42:08.851155 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.851142 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:42:08.851309 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.851293 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal" Oct 30 00:42:08.851351 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.851326 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:42:08.851905 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.851890 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-206.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:42:08.851977 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.851913 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-206.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:42:08.851977 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.851891 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-206.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:42:08.851977 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.851963 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-206.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:42:08.851977 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.851924 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-206.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:42:08.852091 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.851977 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-206.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:42:08.854097 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.854083 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-206.ec2.internal" Oct 30 00:42:08.854151 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.854108 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:42:08.854788 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.854754 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-206.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:42:08.854853 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.854803 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-206.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:42:08.854853 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.854814 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-206.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:42:08.873292 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:08.873276 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-206.ec2.internal\" not found" node="ip-10-0-140-206.ec2.internal" Oct 30 00:42:08.875364 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:08.875350 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-206.ec2.internal\" not found" node="ip-10-0-140-206.ec2.internal" Oct 30 00:42:08.880258 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:08.880244 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-206.ec2.internal\" not found" Oct 30 00:42:08.926683 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.926610 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91d0355fca0e17295f697c41968d06d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal\" (UID: \"91d0355fca0e17295f697c41968d06d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal" Oct 30 00:42:08.926683 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.926647 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91d0355fca0e17295f697c41968d06d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal\" (UID: \"91d0355fca0e17295f697c41968d06d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal" Oct 30 00:42:08.926683 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:08.926671 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b175f0644888f98ec9cdc35355eb1b8-config\") pod \"kube-apiserver-proxy-ip-10-0-140-206.ec2.internal\" (UID: \"3b175f0644888f98ec9cdc35355eb1b8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-206.ec2.internal" Oct 30 00:42:08.980612 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:08.980584 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-206.ec2.internal\" not found" Oct 30 00:42:09.027710 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.027683 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91d0355fca0e17295f697c41968d06d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal\" (UID: \"91d0355fca0e17295f697c41968d06d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal" Oct 30 00:42:09.027799 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.027715 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91d0355fca0e17295f697c41968d06d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal\" (UID: \"91d0355fca0e17295f697c41968d06d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal" Oct 30 00:42:09.027799 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.027734 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b175f0644888f98ec9cdc35355eb1b8-config\") pod \"kube-apiserver-proxy-ip-10-0-140-206.ec2.internal\" (UID: \"3b175f0644888f98ec9cdc35355eb1b8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-206.ec2.internal" Oct 30 00:42:09.027799 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.027791 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91d0355fca0e17295f697c41968d06d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal\" (UID: \"91d0355fca0e17295f697c41968d06d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal" Oct 30 00:42:09.027923 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.027808 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b175f0644888f98ec9cdc35355eb1b8-config\") pod \"kube-apiserver-proxy-ip-10-0-140-206.ec2.internal\" (UID: \"3b175f0644888f98ec9cdc35355eb1b8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-206.ec2.internal" Oct 30 00:42:09.027923 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.027831 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91d0355fca0e17295f697c41968d06d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal\" (UID: \"91d0355fca0e17295f697c41968d06d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal" Oct 30 00:42:09.081991 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:09.081942 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-206.ec2.internal\" not found" Oct 30 00:42:09.177069 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.176997 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal" Oct 30 00:42:09.178195 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.178178 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-206.ec2.internal" Oct 30 00:42:09.182329 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:09.182300 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-206.ec2.internal\" not found" Oct 30 00:42:09.283139 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:09.283096 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-206.ec2.internal\" not found" Oct 30 00:42:09.347660 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.347631 2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:42:09.418326 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.418295 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 00:42:09.418841 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.418444 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 00:42:09.418841 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.418477 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 00:42:09.418841 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.418484 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 00:42:09.423919 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.423899 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal" Oct 30 00:42:09.463647 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.463549 2573 apiserver.go:52] "Watching apiserver" Oct 30 00:42:09.511538 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.511508 2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 00:42:09.511918 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.511889 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-c4b2t","openshift-ovn-kubernetes/ovnkube-node-2nq94","kube-system/global-pull-secret-syncer-xdkdr","openshift-dns/node-resolver-6mjml","openshift-multus/multus-additional-cni-plugins-5rhzn","openshift-multus/network-metrics-daemon-przs7","kube-system/konnectivity-agent-89gxj","openshift-cluster-node-tuning-operator/tuned-pnxhn","openshift-network-operator/iptables-alerter-xpngc","openshift-image-registry/node-ca-g9ftg","openshift-multus/multus-xmr6l","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst"] Oct 30 00:42:09.514614 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.514591 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:09.514725 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:09.514677 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c4b2t" podUID="b3f55f18-4143-4806-b9e4-989e153f7e38" Oct 30 00:42:09.516689 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.516649 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 00:37:09 +0000 UTC" deadline="2028-07-21 15:44:04.62685749 +0000 UTC" Oct 30 00:42:09.516689 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.516687 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15135h1m55.110172139s" Oct 30 00:42:09.516890 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.516879 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.519240 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.519219 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:09.519335 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:09.519279 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdkdr" podUID="b0661ca3-6f0a-4744-9d50-f2baa1a80b45" Oct 30 00:42:09.521484 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.521466 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6mjml" Oct 30 00:42:09.523522 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.523506 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:42:09.523618 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.523595 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-206.ec2.internal" Oct 30 00:42:09.523766 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.523750 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:42:09.525907 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.525892 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:09.526013 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:09.525952 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-przs7" podUID="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" Oct 30 00:42:09.528225 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.528211 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-89gxj" Oct 30 00:42:09.530512 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.530491 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-node-log\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.530578 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.530517 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/995884d0-cb39-4c07-9dbe-5cd3529d5908-tmp-dir\") pod \"node-resolver-6mjml\" (UID: \"995884d0-cb39-4c07-9dbe-5cd3529d5908\") " pod="openshift-dns/node-resolver-6mjml" Oct 30 00:42:09.530578 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.530535 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlnvv\" (UniqueName: \"kubernetes.io/projected/49f04f3b-49d5-4c22-88a2-e33e106853ce-kube-api-access-tlnvv\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:42:09.530653 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.530611 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-run-netns\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.530653 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.530633 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-var-lib-openvswitch\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.530741 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.530672 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49f04f3b-49d5-4c22-88a2-e33e106853ce-cnibin\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:42:09.530741 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.530700 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.530861 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.530738 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x2hb\" (UniqueName: \"kubernetes.io/projected/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-kube-api-access-9x2hb\") pod \"network-metrics-daemon-przs7\" (UID: \"1c2f1e93-f19e-4c0f-944e-7f52261f6e81\") " pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:09.530861 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.530812 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-systemd-units\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.530861 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.530852 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-run-systemd\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.531017 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.530899 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-etc-openvswitch\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.531017 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.530922 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-run-ovn\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.531017 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.530949 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-run-ovn-kubernetes\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.531017 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.530982 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/995884d0-cb39-4c07-9dbe-5cd3529d5908-hosts-file\") pod \"node-resolver-6mjml\" (UID: \"995884d0-cb39-4c07-9dbe-5cd3529d5908\") " pod="openshift-dns/node-resolver-6mjml" Oct 30 00:42:09.531174 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.531046 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/994c018e-e6c5-4d21-adac-9540676a67c0-ovnkube-script-lib\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.531174 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.531106 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r7kr\" (UniqueName: \"kubernetes.io/projected/994c018e-e6c5-4d21-adac-9540676a67c0-kube-api-access-9r7kr\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.531174 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.531142 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-kubelet-config\") pod \"global-pull-secret-syncer-xdkdr\" (UID: \"b0661ca3-6f0a-4744-9d50-f2baa1a80b45\") " pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:09.531174 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.531168 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-dbus\") pod \"global-pull-secret-syncer-xdkdr\" (UID: \"b0661ca3-6f0a-4744-9d50-f2baa1a80b45\") " pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:09.531372 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.531214 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-original-pull-secret\") pod \"global-pull-secret-syncer-xdkdr\" (UID: \"b0661ca3-6f0a-4744-9d50-f2baa1a80b45\") " pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:09.531372 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.531263 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49f04f3b-49d5-4c22-88a2-e33e106853ce-system-cni-dir\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:42:09.531372 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.531318 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49f04f3b-49d5-4c22-88a2-e33e106853ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:42:09.531372 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.531350 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4gkr\" (UniqueName: \"kubernetes.io/projected/b3f55f18-4143-4806-b9e4-989e153f7e38-kube-api-access-m4gkr\") pod \"network-check-target-c4b2t\" (UID: \"b3f55f18-4143-4806-b9e4-989e153f7e38\") " pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:09.531555 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.531433 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.531555 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.531459 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/994c018e-e6c5-4d21-adac-9540676a67c0-ovn-node-metrics-cert\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.531555 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.531480 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb9nw\" (UniqueName: \"kubernetes.io/projected/995884d0-cb39-4c07-9dbe-5cd3529d5908-kube-api-access-fb9nw\") pod \"node-resolver-6mjml\" (UID: \"995884d0-cb39-4c07-9dbe-5cd3529d5908\") " pod="openshift-dns/node-resolver-6mjml" Oct 30 00:42:09.531555 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.531499 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-metrics-certs\") pod \"network-metrics-daemon-przs7\" (UID: \"1c2f1e93-f19e-4c0f-944e-7f52261f6e81\") " pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:09.531797 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.531565 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-kubelet\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.531797 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.531609 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-cni-netd\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.531797 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.531640 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49f04f3b-49d5-4c22-88a2-e33e106853ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:42:09.531797 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.531665 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49f04f3b-49d5-4c22-88a2-e33e106853ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:42:09.531797 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.531689 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-slash\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.531797 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.531712 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/994c018e-e6c5-4d21-adac-9540676a67c0-env-overrides\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.531797 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.531729 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49f04f3b-49d5-4c22-88a2-e33e106853ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:42:09.531797 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.531788 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-cni-bin\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.532367 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.531868 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-run-openvswitch\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.532367 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.531893 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-log-socket\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.532367 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.531917 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/994c018e-e6c5-4d21-adac-9540676a67c0-ovnkube-config\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.533276 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.533257 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xpngc" Oct 30 00:42:09.534006 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.533987 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 00:42:09.534108 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.533987 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 00:42:09.534108 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.533992 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 00:42:09.535705 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.535688 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g9ftg" Oct 30 00:42:09.538089 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.538065 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.538229 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.538214 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 00:42:09.538912 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.538844 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 00:42:09.538912 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.538873 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 00:42:09.538912 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.538905 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 00:42:09.540951 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.539028 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 00:42:09.540951 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.539036 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hp8gq\"" Oct 30 00:42:09.540951 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.539154 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-stkll\"" Oct 30 00:42:09.540951 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.539991 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 00:42:09.541904 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.541884 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kfrd4\"" Oct 30 00:42:09.541976 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.541962 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal"] Oct 30 00:42:09.542045 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.542035 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:42:09.545336 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.545318 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 00:42:09.545400 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.545378 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:42:09.545488 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.545473 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-206.ec2.internal"] Oct 30 00:42:09.622002 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.621982 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 00:42:09.624824 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.624809 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 00:42:09.632654 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.632633 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-run-openvswitch\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.632736 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.632662 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-log-socket\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.632736 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.632684 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-run-k8s-cni-cncf-io\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.632736 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.632700 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-var-lib-cni-bin\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.632736 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.632719 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-sys\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.632897 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.632766 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-run-openvswitch\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.632897 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.632777 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/995884d0-cb39-4c07-9dbe-5cd3529d5908-tmp-dir\") pod \"node-resolver-6mjml\" (UID: \"995884d0-cb39-4c07-9dbe-5cd3529d5908\") " pod="openshift-dns/node-resolver-6mjml" Oct 30 00:42:09.632897 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.632756 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-log-socket\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.633001 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.632904 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-var-lib-openvswitch\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.633001 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.632928 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-sys-fs\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:42:09.633001 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.632964 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-var-lib-openvswitch\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.633001 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.632974 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-systemd\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.633001 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.632996 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-var-lib-kubelet\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.633171 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633025 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49f04f3b-49d5-4c22-88a2-e33e106853ce-cnibin\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:42:09.633171 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633061 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49f04f3b-49d5-4c22-88a2-e33e106853ce-cnibin\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:42:09.633171 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633095 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x2hb\" (UniqueName: \"kubernetes.io/projected/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-kube-api-access-9x2hb\") pod \"network-metrics-daemon-przs7\" (UID: \"1c2f1e93-f19e-4c0f-944e-7f52261f6e81\") " pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:09.633171 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633127 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-run-systemd\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.633171 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633153 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7127ca6b-3962-47a9-9089-07bcba277bc7-host-slash\") pod \"iptables-alerter-xpngc\" (UID: \"7127ca6b-3962-47a9-9089-07bcba277bc7\") " pod="openshift-network-operator/iptables-alerter-xpngc" Oct 30 00:42:09.633342 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633171 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-run-netns\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.633342 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633170 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-run-systemd\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.633342 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633214 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:42:09.633342 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633245 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/995884d0-cb39-4c07-9dbe-5cd3529d5908-hosts-file\") pod \"node-resolver-6mjml\" (UID: \"995884d0-cb39-4c07-9dbe-5cd3529d5908\") " pod="openshift-dns/node-resolver-6mjml" Oct 30 00:42:09.633342 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633285 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-original-pull-secret\") pod \"global-pull-secret-syncer-xdkdr\" (UID: \"b0661ca3-6f0a-4744-9d50-f2baa1a80b45\") " pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:09.633342 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633291 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/995884d0-cb39-4c07-9dbe-5cd3529d5908-hosts-file\") pod \"node-resolver-6mjml\" (UID: \"995884d0-cb39-4c07-9dbe-5cd3529d5908\") " pod="openshift-dns/node-resolver-6mjml" Oct 30 00:42:09.633561 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:09.633375 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:09.633561 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633381 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-cnibin\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.633561 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633399 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/da571014-afa3-4a40-a083-dc708bf83ba3-multus-daemon-config\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.633561 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633423 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49f04f3b-49d5-4c22-88a2-e33e106853ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:42:09.633561 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:09.633445 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-original-pull-secret podName:b0661ca3-6f0a-4744-9d50-f2baa1a80b45 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:10.133421578 +0000 UTC m=+2.007152727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-original-pull-secret") pod "global-pull-secret-syncer-xdkdr" (UID: "b0661ca3-6f0a-4744-9d50-f2baa1a80b45") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:09.633561 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633495 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.633561 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633515 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/995884d0-cb39-4c07-9dbe-5cd3529d5908-tmp-dir\") pod \"node-resolver-6mjml\" (UID: \"995884d0-cb39-4c07-9dbe-5cd3529d5908\") " pod="openshift-dns/node-resolver-6mjml" Oct 30 00:42:09.633561 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633527 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/994c018e-e6c5-4d21-adac-9540676a67c0-ovn-node-metrics-cert\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.633561 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633557 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.634024 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633649 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndpj2\" (UniqueName: \"kubernetes.io/projected/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-kube-api-access-ndpj2\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.634024 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633793 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49f04f3b-49d5-4c22-88a2-e33e106853ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:42:09.634024 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633829 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49f04f3b-49d5-4c22-88a2-e33e106853ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:42:09.634024 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633865 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-kubernetes\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.634024 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633891 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-tuned\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.634024 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633907 2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 00:42:09.634024 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633921 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49f04f3b-49d5-4c22-88a2-e33e106853ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:42:09.634024 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633962 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-system-cni-dir\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.634024 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633990 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-multus-cni-dir\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.634024 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633992 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49f04f3b-49d5-4c22-88a2-e33e106853ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:42:09.634024 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.633985 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49f04f3b-49d5-4c22-88a2-e33e106853ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:42:09.634581 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634062 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-multus-socket-dir-parent\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.634581 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634083 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-multus-conf-dir\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.634581 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634104 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-etc-kubernetes\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.634581 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634134 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-host\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.634581 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634157 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w48mv\" (UniqueName: \"kubernetes.io/projected/5dff88d3-2e35-40e9-8aea-208519485adc-kube-api-access-w48mv\") pod \"node-ca-g9ftg\" (UID: \"5dff88d3-2e35-40e9-8aea-208519485adc\") " pod="openshift-image-registry/node-ca-g9ftg" Oct 30 00:42:09.634581 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634189 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/994c018e-e6c5-4d21-adac-9540676a67c0-ovnkube-config\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.634581 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634231 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-registration-dir\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:42:09.634581 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634251 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-sysctl-conf\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.634581 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634267 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-lib-modules\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.634581 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634288 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5dff88d3-2e35-40e9-8aea-208519485adc-host\") pod \"node-ca-g9ftg\" (UID: \"5dff88d3-2e35-40e9-8aea-208519485adc\") " pod="openshift-image-registry/node-ca-g9ftg" Oct 30 00:42:09.634581 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634344 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-node-log\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.634581 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634362 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlnvv\" (UniqueName: \"kubernetes.io/projected/49f04f3b-49d5-4c22-88a2-e33e106853ce-kube-api-access-tlnvv\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:42:09.634581 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634399 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-run-netns\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.634581 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634408 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-node-log\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.634581 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634431 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49f04f3b-49d5-4c22-88a2-e33e106853ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:42:09.634581 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634445 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/271e26db-6a8b-4e46-afb7-139cbc57c1e1-agent-certs\") pod \"konnectivity-agent-89gxj\" (UID: \"271e26db-6a8b-4e46-afb7-139cbc57c1e1\") " pod="kube-system/konnectivity-agent-89gxj" Oct 30 00:42:09.634581 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634492 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-systemd-units\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.635481 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634509 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-etc-openvswitch\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.635481 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634509 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-run-netns\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.635481 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634524 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-run-ovn\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.635481 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634556 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-run-ovn-kubernetes\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.635481 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634560 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-run-ovn\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.635481 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634575 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-systemd-units\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.635481 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634599 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-etc-openvswitch\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.635481 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634608 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-run-ovn-kubernetes\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.635481 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634625 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r7kr\" (UniqueName: \"kubernetes.io/projected/994c018e-e6c5-4d21-adac-9540676a67c0-kube-api-access-9r7kr\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.635481 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634642 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-dbus\") pod \"global-pull-secret-syncer-xdkdr\" (UID: \"b0661ca3-6f0a-4744-9d50-f2baa1a80b45\") " pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:09.635481 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634741 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/271e26db-6a8b-4e46-afb7-139cbc57c1e1-konnectivity-ca\") pod \"konnectivity-agent-89gxj\" (UID: \"271e26db-6a8b-4e46-afb7-139cbc57c1e1\") " pod="kube-system/konnectivity-agent-89gxj" Oct 30 00:42:09.635481 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634752 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-dbus\") pod \"global-pull-secret-syncer-xdkdr\" (UID: \"b0661ca3-6f0a-4744-9d50-f2baa1a80b45\") " pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:09.635481 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634789 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/994c018e-e6c5-4d21-adac-9540676a67c0-ovnkube-script-lib\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.635481 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634835 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-kubelet-config\") pod \"global-pull-secret-syncer-xdkdr\" (UID: \"b0661ca3-6f0a-4744-9d50-f2baa1a80b45\") " pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:09.635481 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634869 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj6jk\" (UniqueName: \"kubernetes.io/projected/7127ca6b-3962-47a9-9089-07bcba277bc7-kube-api-access-hj6jk\") pod \"iptables-alerter-xpngc\" (UID: \"7127ca6b-3962-47a9-9089-07bcba277bc7\") " pod="openshift-network-operator/iptables-alerter-xpngc" Oct 30 00:42:09.635481 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634890 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-var-lib-kubelet\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.635481 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634904 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/994c018e-e6c5-4d21-adac-9540676a67c0-ovnkube-config\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.636078 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634923 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-hostroot\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.636078 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634967 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-kubelet-config\") pod \"global-pull-secret-syncer-xdkdr\" (UID: \"b0661ca3-6f0a-4744-9d50-f2baa1a80b45\") " pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:09.636078 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.634973 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-device-dir\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:42:09.636078 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635039 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-etc-selinux\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:42:09.636078 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635058 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49f04f3b-49d5-4c22-88a2-e33e106853ce-system-cni-dir\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:42:09.636078 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635077 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4gkr\" (UniqueName: \"kubernetes.io/projected/b3f55f18-4143-4806-b9e4-989e153f7e38-kube-api-access-m4gkr\") pod \"network-check-target-c4b2t\" (UID: \"b3f55f18-4143-4806-b9e4-989e153f7e38\") " pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:09.636078 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635112 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49f04f3b-49d5-4c22-88a2-e33e106853ce-system-cni-dir\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:42:09.636078 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635137 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7127ca6b-3962-47a9-9089-07bcba277bc7-iptables-alerter-script\") pod \"iptables-alerter-xpngc\" (UID: \"7127ca6b-3962-47a9-9089-07bcba277bc7\") " pod="openshift-network-operator/iptables-alerter-xpngc" Oct 30 00:42:09.636078 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635161 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da571014-afa3-4a40-a083-dc708bf83ba3-cni-binary-copy\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.636078 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635179 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-socket-dir\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:42:09.636078 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635199 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-modprobe-d\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.636078 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635231 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-sysconfig\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.636078 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635265 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-run\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.636078 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635287 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb9nw\" (UniqueName: \"kubernetes.io/projected/995884d0-cb39-4c07-9dbe-5cd3529d5908-kube-api-access-fb9nw\") pod \"node-resolver-6mjml\" (UID: \"995884d0-cb39-4c07-9dbe-5cd3529d5908\") " pod="openshift-dns/node-resolver-6mjml" Oct 30 00:42:09.636078 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635334 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-metrics-certs\") pod \"network-metrics-daemon-przs7\" (UID: \"1c2f1e93-f19e-4c0f-944e-7f52261f6e81\") " pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:09.636078 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635362 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-kubelet\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.636078 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635380 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/994c018e-e6c5-4d21-adac-9540676a67c0-ovnkube-script-lib\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.636571 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635387 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-cni-netd\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.636571 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:09.635412 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:09.636571 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635422 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-cni-netd\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.636571 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635438 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-run-multus-certs\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.636571 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635475 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-kubelet\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.636571 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:09.635481 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-metrics-certs podName:1c2f1e93-f19e-4c0f-944e-7f52261f6e81 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:10.135440963 +0000 UTC m=+2.009172109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-metrics-certs") pod "network-metrics-daemon-przs7" (UID: "1c2f1e93-f19e-4c0f-944e-7f52261f6e81") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:09.636571 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635513 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rxlb\" (UniqueName: \"kubernetes.io/projected/da571014-afa3-4a40-a083-dc708bf83ba3-kube-api-access-9rxlb\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.636571 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635538 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l7h2\" (UniqueName: \"kubernetes.io/projected/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-kube-api-access-4l7h2\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:42:09.636571 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635565 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5dff88d3-2e35-40e9-8aea-208519485adc-serviceca\") pod \"node-ca-g9ftg\" (UID: \"5dff88d3-2e35-40e9-8aea-208519485adc\") " pod="openshift-image-registry/node-ca-g9ftg" Oct 30 00:42:09.636571 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635589 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-slash\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.636571 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635616 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/994c018e-e6c5-4d21-adac-9540676a67c0-env-overrides\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.636571 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635633 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-var-lib-cni-multus\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.636571 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635670 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-sysctl-d\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.636571 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635689 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-slash\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.636571 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635751 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-tmp\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.636571 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635794 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-cni-bin\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.636571 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635834 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-cni-bin\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.637093 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.635994 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/994c018e-e6c5-4d21-adac-9540676a67c0-env-overrides\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.637528 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.637504 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/994c018e-e6c5-4d21-adac-9540676a67c0-ovn-node-metrics-cert\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.642882 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.642846 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:42:09.642882 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.642864 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 00:42:09.643001 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.642932 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 00:42:09.643001 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.642956 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 00:42:09.643253 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.643223 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 00:42:09.643253 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.643230 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 00:42:09.643253 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.643242 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 00:42:09.643253 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.643230 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 00:42:09.643472 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.643286 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 00:42:09.643472 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.643298 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 00:42:09.643472 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.643287 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 00:42:09.643472 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.643375 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 00:42:09.643472 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.643405 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 00:42:09.643472 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.643449 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 00:42:09.644101 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.644083 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:42:09.644436 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.644416 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49f04f3b-49d5-4c22-88a2-e33e106853ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:42:09.647832 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.647571 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hnsxr\"" Oct 30 00:42:09.647832 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.647583 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 00:42:09.647832 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.647609 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8tgxx\"" Oct 30 00:42:09.647832 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.647631 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7ppwp\"" Oct 30 00:42:09.647832 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.647662 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v7mtf\"" Oct 30 00:42:09.647832 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.647669 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cd6zv\"" Oct 30 00:42:09.647832 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.647690 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tlfwv\"" Oct 30 00:42:09.648239 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.648214 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 00:42:09.656568 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:09.654669 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:42:09.656568 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:09.654701 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:42:09.656568 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:09.654715 2573 projected.go:196] Error preparing data for projected volume kube-api-access-m4gkr for pod openshift-network-diagnostics/network-check-target-c4b2t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:09.656568 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:09.654796 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3f55f18-4143-4806-b9e4-989e153f7e38-kube-api-access-m4gkr podName:b3f55f18-4143-4806-b9e4-989e153f7e38 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:10.154775483 +0000 UTC m=+2.028506633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m4gkr" (UniqueName: "kubernetes.io/projected/b3f55f18-4143-4806-b9e4-989e153f7e38-kube-api-access-m4gkr") pod "network-check-target-c4b2t" (UID: "b3f55f18-4143-4806-b9e4-989e153f7e38") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:09.656568 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.656490 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x2hb\" (UniqueName: \"kubernetes.io/projected/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-kube-api-access-9x2hb\") pod \"network-metrics-daemon-przs7\" (UID: \"1c2f1e93-f19e-4c0f-944e-7f52261f6e81\") " pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:09.656892 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.656703 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb9nw\" (UniqueName: \"kubernetes.io/projected/995884d0-cb39-4c07-9dbe-5cd3529d5908-kube-api-access-fb9nw\") pod \"node-resolver-6mjml\" (UID: \"995884d0-cb39-4c07-9dbe-5cd3529d5908\") " pod="openshift-dns/node-resolver-6mjml" Oct 30 00:42:09.657549 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.657526 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r7kr\" (UniqueName: \"kubernetes.io/projected/994c018e-e6c5-4d21-adac-9540676a67c0-kube-api-access-9r7kr\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.658444 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.658424 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlnvv\" (UniqueName: \"kubernetes.io/projected/49f04f3b-49d5-4c22-88a2-e33e106853ce-kube-api-access-tlnvv\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:42:09.677653 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.677628 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 00:42:09.677754 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.677695 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lhl68" Oct 30 00:42:09.737086 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737054 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-var-lib-cni-multus\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.737086 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737092 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-sysctl-d\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.737376 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737111 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-tmp\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.737376 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737129 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-run-k8s-cni-cncf-io\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.737376 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737200 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-var-lib-cni-multus\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.737376 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737284 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-sysctl-d\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.737376 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737290 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-var-lib-cni-bin\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.737376 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737348 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-var-lib-cni-bin\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.737376 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737372 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-run-k8s-cni-cncf-io\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.737693 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737381 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-sys\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.737693 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737416 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-sys-fs\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:42:09.737693 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737441 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-sys\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.737693 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737446 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-systemd\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.737693 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737493 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-systemd\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.737693 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737501 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-var-lib-kubelet\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.737693 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737504 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-sys-fs\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:42:09.737693 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737539 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7127ca6b-3962-47a9-9089-07bcba277bc7-host-slash\") pod \"iptables-alerter-xpngc\" (UID: \"7127ca6b-3962-47a9-9089-07bcba277bc7\") " pod="openshift-network-operator/iptables-alerter-xpngc" Oct 30 00:42:09.737693 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737570 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-run-netns\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.737693 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737579 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-var-lib-kubelet\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.737693 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737600 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:42:09.737693 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737613 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7127ca6b-3962-47a9-9089-07bcba277bc7-host-slash\") pod \"iptables-alerter-xpngc\" (UID: \"7127ca6b-3962-47a9-9089-07bcba277bc7\") " pod="openshift-network-operator/iptables-alerter-xpngc" Oct 30 00:42:09.737693 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737624 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-run-netns\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.737693 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737659 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-cnibin\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.737693 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737664 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:42:09.737693 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737694 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/da571014-afa3-4a40-a083-dc708bf83ba3-multus-daemon-config\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.738209 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737730 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndpj2\" (UniqueName: \"kubernetes.io/projected/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-kube-api-access-ndpj2\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.738209 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737753 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-cnibin\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.738209 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737755 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-kubernetes\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.738209 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737820 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-kubernetes\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.738209 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737821 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-tuned\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.738209 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737855 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-system-cni-dir\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.738209 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737889 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-multus-cni-dir\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.738209 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737915 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-multus-socket-dir-parent\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.738209 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737938 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-multus-conf-dir\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.738209 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737963 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-etc-kubernetes\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.738209 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.737986 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-host\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.738209 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.738015 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w48mv\" (UniqueName: \"kubernetes.io/projected/5dff88d3-2e35-40e9-8aea-208519485adc-kube-api-access-w48mv\") pod \"node-ca-g9ftg\" (UID: \"5dff88d3-2e35-40e9-8aea-208519485adc\") " pod="openshift-image-registry/node-ca-g9ftg" Oct 30 00:42:09.738209 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.738042 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-registration-dir\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:42:09.738209 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.738068 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-sysctl-conf\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.738209 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.738094 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-lib-modules\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.738209 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.738120 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5dff88d3-2e35-40e9-8aea-208519485adc-host\") pod \"node-ca-g9ftg\" (UID: \"5dff88d3-2e35-40e9-8aea-208519485adc\") " pod="openshift-image-registry/node-ca-g9ftg" Oct 30 00:42:09.738209 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.738149 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/271e26db-6a8b-4e46-afb7-139cbc57c1e1-agent-certs\") pod \"konnectivity-agent-89gxj\" (UID: \"271e26db-6a8b-4e46-afb7-139cbc57c1e1\") " pod="kube-system/konnectivity-agent-89gxj" Oct 30 00:42:09.738209 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.738191 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/271e26db-6a8b-4e46-afb7-139cbc57c1e1-konnectivity-ca\") pod \"konnectivity-agent-89gxj\" (UID: \"271e26db-6a8b-4e46-afb7-139cbc57c1e1\") " pod="kube-system/konnectivity-agent-89gxj" Oct 30 00:42:09.739160 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.738235 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj6jk\" (UniqueName: \"kubernetes.io/projected/7127ca6b-3962-47a9-9089-07bcba277bc7-kube-api-access-hj6jk\") pod \"iptables-alerter-xpngc\" (UID: \"7127ca6b-3962-47a9-9089-07bcba277bc7\") " pod="openshift-network-operator/iptables-alerter-xpngc" Oct 30 00:42:09.739160 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.738264 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-var-lib-kubelet\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.739160 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.738309 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-hostroot\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.739160 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.738335 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-device-dir\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:42:09.739160 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.738361 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-etc-selinux\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:42:09.739160 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.738403 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7127ca6b-3962-47a9-9089-07bcba277bc7-iptables-alerter-script\") pod \"iptables-alerter-xpngc\" (UID: \"7127ca6b-3962-47a9-9089-07bcba277bc7\") " pod="openshift-network-operator/iptables-alerter-xpngc" Oct 30 00:42:09.739160 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.738430 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da571014-afa3-4a40-a083-dc708bf83ba3-cni-binary-copy\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.739160 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.738456 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-socket-dir\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:42:09.739160 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.738494 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-modprobe-d\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.739160 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.738521 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-sysconfig\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.739160 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.738560 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-run\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.739160 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.738600 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-run-multus-certs\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.739160 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.738626 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rxlb\" (UniqueName: \"kubernetes.io/projected/da571014-afa3-4a40-a083-dc708bf83ba3-kube-api-access-9rxlb\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.739160 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.738656 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l7h2\" (UniqueName: \"kubernetes.io/projected/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-kube-api-access-4l7h2\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:42:09.739160 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.738681 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5dff88d3-2e35-40e9-8aea-208519485adc-serviceca\") pod \"node-ca-g9ftg\" (UID: \"5dff88d3-2e35-40e9-8aea-208519485adc\") " pod="openshift-image-registry/node-ca-g9ftg" Oct 30 00:42:09.739160 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.738826 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/da571014-afa3-4a40-a083-dc708bf83ba3-multus-daemon-config\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.739160 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.739082 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-socket-dir\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:42:09.740131 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.739156 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-var-lib-kubelet\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.740131 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.739163 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-system-cni-dir\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.740131 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.739209 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-hostroot\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.740131 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.739229 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5dff88d3-2e35-40e9-8aea-208519485adc-serviceca\") pod \"node-ca-g9ftg\" (UID: \"5dff88d3-2e35-40e9-8aea-208519485adc\") " pod="openshift-image-registry/node-ca-g9ftg" Oct 30 00:42:09.740131 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.739271 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-device-dir\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:42:09.740131 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.739303 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-multus-cni-dir\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.740131 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.739361 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-etc-selinux\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:42:09.740131 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.739368 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-multus-socket-dir-parent\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.740131 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.739341 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-modprobe-d\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.740131 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.739418 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-sysconfig\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.740131 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.739458 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-run\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.740131 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.739505 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-multus-conf-dir\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.740131 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.739549 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-etc-kubernetes\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.740131 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.739593 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-host\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.740131 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.739758 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-registration-dir\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:42:09.740131 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.739890 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5dff88d3-2e35-40e9-8aea-208519485adc-host\") pod \"node-ca-g9ftg\" (UID: \"5dff88d3-2e35-40e9-8aea-208519485adc\") " pod="openshift-image-registry/node-ca-g9ftg" Oct 30 00:42:09.740131 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.739898 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-run-multus-certs\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.740131 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.739895 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-lib-modules\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.741014 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.739913 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-sysctl-conf\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.741014 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.740338 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-tmp\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.741014 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.740489 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7127ca6b-3962-47a9-9089-07bcba277bc7-iptables-alerter-script\") pod \"iptables-alerter-xpngc\" (UID: \"7127ca6b-3962-47a9-9089-07bcba277bc7\") " pod="openshift-network-operator/iptables-alerter-xpngc" Oct 30 00:42:09.741014 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.740609 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da571014-afa3-4a40-a083-dc708bf83ba3-cni-binary-copy\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.741014 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.740766 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-tuned\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.741014 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.740899 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/271e26db-6a8b-4e46-afb7-139cbc57c1e1-konnectivity-ca\") pod \"konnectivity-agent-89gxj\" (UID: \"271e26db-6a8b-4e46-afb7-139cbc57c1e1\") " pod="kube-system/konnectivity-agent-89gxj" Oct 30 00:42:09.742825 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.742765 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/271e26db-6a8b-4e46-afb7-139cbc57c1e1-agent-certs\") pod \"konnectivity-agent-89gxj\" (UID: \"271e26db-6a8b-4e46-afb7-139cbc57c1e1\") " pod="kube-system/konnectivity-agent-89gxj" Oct 30 00:42:09.761083 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.761056 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndpj2\" (UniqueName: \"kubernetes.io/projected/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-kube-api-access-ndpj2\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.761521 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.761499 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l7h2\" (UniqueName: \"kubernetes.io/projected/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-kube-api-access-4l7h2\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:42:09.761720 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.761698 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj6jk\" (UniqueName: \"kubernetes.io/projected/7127ca6b-3962-47a9-9089-07bcba277bc7-kube-api-access-hj6jk\") pod \"iptables-alerter-xpngc\" (UID: \"7127ca6b-3962-47a9-9089-07bcba277bc7\") " pod="openshift-network-operator/iptables-alerter-xpngc" Oct 30 00:42:09.761906 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.761887 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rxlb\" (UniqueName: \"kubernetes.io/projected/da571014-afa3-4a40-a083-dc708bf83ba3-kube-api-access-9rxlb\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.762441 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.762422 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w48mv\" (UniqueName: \"kubernetes.io/projected/5dff88d3-2e35-40e9-8aea-208519485adc-kube-api-access-w48mv\") pod \"node-ca-g9ftg\" (UID: \"5dff88d3-2e35-40e9-8aea-208519485adc\") " pod="openshift-image-registry/node-ca-g9ftg" Oct 30 00:42:09.837939 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:09.837897 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b175f0644888f98ec9cdc35355eb1b8.slice/crio-8aedcc0431068aad1e5868f70d8904e299ca29410447c75871583ef000efb149 WatchSource:0}: Error finding container 8aedcc0431068aad1e5868f70d8904e299ca29410447c75871583ef000efb149: Status 404 returned error can't find the container with id 8aedcc0431068aad1e5868f70d8904e299ca29410447c75871583ef000efb149 Oct 30 00:42:09.846491 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.846471 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:42:09.849623 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:09.849596 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d0355fca0e17295f697c41968d06d7.slice/crio-1a21ea02e80d4f6d786dbd8b2d11fb89a274e7c3ff0c3b317588934510b97e06 WatchSource:0}: Error finding container 1a21ea02e80d4f6d786dbd8b2d11fb89a274e7c3ff0c3b317588934510b97e06: Status 404 returned error can't find the container with id 1a21ea02e80d4f6d786dbd8b2d11fb89a274e7c3ff0c3b317588934510b97e06 Oct 30 00:42:09.860463 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.860444 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6mjml" Oct 30 00:42:09.867660 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.867642 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:09.870190 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:09.870166 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod995884d0_cb39_4c07_9dbe_5cd3529d5908.slice/crio-cd71fcdd9fcb74d62ca6fd43f9446fd30908000a64453e19d6f2e8de7bafcadd WatchSource:0}: Error finding container cd71fcdd9fcb74d62ca6fd43f9446fd30908000a64453e19d6f2e8de7bafcadd: Status 404 returned error can't find the container with id cd71fcdd9fcb74d62ca6fd43f9446fd30908000a64453e19d6f2e8de7bafcadd Oct 30 00:42:09.896517 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.896495 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:42:09.902737 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.902723 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-89gxj" Oct 30 00:42:09.905464 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.905437 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:42:09.906128 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:09.906109 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f04f3b_49d5_4c22_88a2_e33e106853ce.slice/crio-839ff87247ef7ca34d8a7fecffcd2d142d915609c291cc4c257adcda2241eb00 WatchSource:0}: Error finding container 839ff87247ef7ca34d8a7fecffcd2d142d915609c291cc4c257adcda2241eb00: Status 404 returned error can't find the container with id 839ff87247ef7ca34d8a7fecffcd2d142d915609c291cc4c257adcda2241eb00 Oct 30 00:42:09.930230 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.930209 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xpngc" Oct 30 00:42:09.939169 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:09.939143 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7127ca6b_3962_47a9_9089_07bcba277bc7.slice/crio-b76808fc47c412c7ba94d1e61361c17514fc485ba206f74e99d4d3584f101901 WatchSource:0}: Error finding container b76808fc47c412c7ba94d1e61361c17514fc485ba206f74e99d4d3584f101901: Status 404 returned error can't find the container with id b76808fc47c412c7ba94d1e61361c17514fc485ba206f74e99d4d3584f101901 Oct 30 00:42:09.946986 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.946969 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g9ftg" Oct 30 00:42:09.957000 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:09.956978 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dff88d3_2e35_40e9_8aea_208519485adc.slice/crio-641e3ef2e21a72d92a5de7c37c8369f43fc34067da733bababa0b8bba76242c1 WatchSource:0}: Error finding container 641e3ef2e21a72d92a5de7c37c8369f43fc34067da733bababa0b8bba76242c1: Status 404 returned error can't find the container with id 641e3ef2e21a72d92a5de7c37c8369f43fc34067da733bababa0b8bba76242c1 Oct 30 00:42:09.969009 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.968990 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xmr6l" Oct 30 00:42:09.979233 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:09.979205 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda571014_afa3_4a40_a083_dc708bf83ba3.slice/crio-c963d0df141bd6ca560142b1b23b521cdf62b24164a3baab88f5318e8503bffd WatchSource:0}: Error finding container c963d0df141bd6ca560142b1b23b521cdf62b24164a3baab88f5318e8503bffd: Status 404 returned error can't find the container with id c963d0df141bd6ca560142b1b23b521cdf62b24164a3baab88f5318e8503bffd Oct 30 00:42:09.984024 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:09.983999 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:42:09.993457 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:42:09.993434 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1c6441_4ebb_48d6_9e09_bf69e431eb57.slice/crio-9ef8f37acadd6669c5af3620533b330d8f1b32b11361b56756eadaf7258e388f WatchSource:0}: Error finding container 9ef8f37acadd6669c5af3620533b330d8f1b32b11361b56756eadaf7258e388f: Status 404 returned error can't find the container with id 9ef8f37acadd6669c5af3620533b330d8f1b32b11361b56756eadaf7258e388f Oct 30 00:42:09.995537 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:09.995515 2573 kuberuntime_manager.go:1707] "Unhandled Error" err="container csi-driver start failed in pod aws-ebs-csi-driver-node-rfmst_openshift-cluster-csi-drivers(1f1c6441-4ebb-48d6-9e09-bf69e431eb57): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Oct 30 00:42:09.996615 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:09.996594 2573 kuberuntime_manager.go:1707] "Unhandled Error" err="container csi-node-driver-registrar start failed in pod aws-ebs-csi-driver-node-rfmst_openshift-cluster-csi-drivers(1f1c6441-4ebb-48d6-9e09-bf69e431eb57): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Oct 30 00:42:09.997795 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:09.997749 2573 kuberuntime_manager.go:1707] "Unhandled Error" err="container csi-liveness-probe start failed in pod aws-ebs-csi-driver-node-rfmst_openshift-cluster-csi-drivers(1f1c6441-4ebb-48d6-9e09-bf69e431eb57): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Oct 30 00:42:09.997878 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:09.997797 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-driver\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"csi-liveness-probe\" with ErrImagePull: \"pull QPS exceeded\"]" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" podUID="1f1c6441-4ebb-48d6-9e09-bf69e431eb57" Oct 30 00:42:10.140595 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:10.140554 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-metrics-certs\") pod \"network-metrics-daemon-przs7\" (UID: \"1c2f1e93-f19e-4c0f-944e-7f52261f6e81\") " pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:10.140595 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:10.140601 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-original-pull-secret\") pod \"global-pull-secret-syncer-xdkdr\" (UID: \"b0661ca3-6f0a-4744-9d50-f2baa1a80b45\") " pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:10.140833 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:10.140704 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:10.140833 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:10.140761 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-original-pull-secret podName:b0661ca3-6f0a-4744-9d50-f2baa1a80b45 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:11.140748144 +0000 UTC m=+3.014479274 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-original-pull-secret") pod "global-pull-secret-syncer-xdkdr" (UID: "b0661ca3-6f0a-4744-9d50-f2baa1a80b45") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:10.140833 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:10.140707 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:10.140833 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:10.140810 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-metrics-certs podName:1c2f1e93-f19e-4c0f-944e-7f52261f6e81 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:11.140786791 +0000 UTC m=+3.014517923 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-metrics-certs") pod "network-metrics-daemon-przs7" (UID: "1c2f1e93-f19e-4c0f-944e-7f52261f6e81") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:10.147723 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:10.147694 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:42:10.241705 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:10.241610 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4gkr\" (UniqueName: \"kubernetes.io/projected/b3f55f18-4143-4806-b9e4-989e153f7e38-kube-api-access-m4gkr\") pod \"network-check-target-c4b2t\" (UID: \"b3f55f18-4143-4806-b9e4-989e153f7e38\") " pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:10.241896 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:10.241777 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:42:10.241896 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:10.241815 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:42:10.241896 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:10.241831 2573 projected.go:196] Error preparing data for projected volume kube-api-access-m4gkr for pod openshift-network-diagnostics/network-check-target-c4b2t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:10.242013 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:10.241901 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3f55f18-4143-4806-b9e4-989e153f7e38-kube-api-access-m4gkr podName:b3f55f18-4143-4806-b9e4-989e153f7e38 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:11.241881397 +0000 UTC m=+3.115612547 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4gkr" (UniqueName: "kubernetes.io/projected/b3f55f18-4143-4806-b9e4-989e153f7e38-kube-api-access-m4gkr") pod "network-check-target-c4b2t" (UID: "b3f55f18-4143-4806-b9e4-989e153f7e38") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:10.307076 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:10.303323 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:42:10.635309 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:10.635241 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:42:10.681904 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:10.681850 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:37:10 +0000 UTC" deadline="2028-08-08 07:22:14.570625604 +0000 UTC" Oct 30 00:42:10.681904 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:10.681903 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15558h40m3.888728709s" Oct 30 00:42:10.763909 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:10.763828 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xmr6l" event={"ID":"da571014-afa3-4a40-a083-dc708bf83ba3","Type":"ContainerStarted","Data":"c963d0df141bd6ca560142b1b23b521cdf62b24164a3baab88f5318e8503bffd"} Oct 30 00:42:10.774245 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:10.774207 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g9ftg" event={"ID":"5dff88d3-2e35-40e9-8aea-208519485adc","Type":"ContainerStarted","Data":"641e3ef2e21a72d92a5de7c37c8369f43fc34067da733bababa0b8bba76242c1"} Oct 30 00:42:10.788728 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:10.788692 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xpngc" event={"ID":"7127ca6b-3962-47a9-9089-07bcba277bc7","Type":"ContainerStarted","Data":"b76808fc47c412c7ba94d1e61361c17514fc485ba206f74e99d4d3584f101901"} Oct 30 00:42:10.805305 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:10.805263 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-89gxj" event={"ID":"271e26db-6a8b-4e46-afb7-139cbc57c1e1","Type":"ContainerStarted","Data":"4cd9efa42b7843ae35ae9d8eadc51084370007512f89c38570b4c4ecc65a59f2"} Oct 30 00:42:10.821694 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:10.821653 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" event={"ID":"994c018e-e6c5-4d21-adac-9540676a67c0","Type":"ContainerStarted","Data":"d9783f5758a10e5b89b5103e04b1261b0cfd92b52c5cf73c558c6b763e53a481"} Oct 30 00:42:10.836595 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:10.836559 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6mjml" event={"ID":"995884d0-cb39-4c07-9dbe-5cd3529d5908","Type":"ContainerStarted","Data":"cd71fcdd9fcb74d62ca6fd43f9446fd30908000a64453e19d6f2e8de7bafcadd"} Oct 30 00:42:10.847398 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:10.847361 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" event={"ID":"1f1c6441-4ebb-48d6-9e09-bf69e431eb57","Type":"ContainerStarted","Data":"9ef8f37acadd6669c5af3620533b330d8f1b32b11361b56756eadaf7258e388f"} Oct 30 00:42:10.870672 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:10.870629 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"csi-driver\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:007ba215236827a40e1b33f3799d289065305ae886c05e51214d74d0ff24d62d\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a5e96bc702bc529b424e6c80f95d7d4fc4fe92f83df4a77c3b47eb3a30fe729\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"csi-liveness-probe\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4db2cbbaecb77e6b3a4e8dd7ee7a4d4fa9ab69787e4330711a3abb1277af2ac\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" podUID="1f1c6441-4ebb-48d6-9e09-bf69e431eb57" Oct 30 00:42:10.872465 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:10.872435 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" event={"ID":"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0","Type":"ContainerStarted","Data":"116f8039d7146d3f834f7a9ec2703f3de229bdb697f2d556b8cca1aa0707b651"} Oct 30 00:42:10.880058 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:10.880032 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rhzn" event={"ID":"49f04f3b-49d5-4c22-88a2-e33e106853ce","Type":"ContainerStarted","Data":"839ff87247ef7ca34d8a7fecffcd2d142d915609c291cc4c257adcda2241eb00"} Oct 30 00:42:10.902542 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:10.902467 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal" event={"ID":"91d0355fca0e17295f697c41968d06d7","Type":"ContainerStarted","Data":"1a21ea02e80d4f6d786dbd8b2d11fb89a274e7c3ff0c3b317588934510b97e06"} Oct 30 00:42:10.916792 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:10.916667 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-206.ec2.internal" event={"ID":"3b175f0644888f98ec9cdc35355eb1b8","Type":"ContainerStarted","Data":"8aedcc0431068aad1e5868f70d8904e299ca29410447c75871583ef000efb149"} Oct 30 00:42:11.149048 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:11.149007 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-metrics-certs\") pod \"network-metrics-daemon-przs7\" (UID: \"1c2f1e93-f19e-4c0f-944e-7f52261f6e81\") " pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:11.149253 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:11.149085 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-original-pull-secret\") pod \"global-pull-secret-syncer-xdkdr\" (UID: \"b0661ca3-6f0a-4744-9d50-f2baa1a80b45\") " pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:11.149327 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:11.149252 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:11.149327 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:11.149314 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-original-pull-secret podName:b0661ca3-6f0a-4744-9d50-f2baa1a80b45 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:13.149296237 +0000 UTC m=+5.023027374 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-original-pull-secret") pod "global-pull-secret-syncer-xdkdr" (UID: "b0661ca3-6f0a-4744-9d50-f2baa1a80b45") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:11.149731 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:11.149708 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:11.149821 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:11.149770 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-metrics-certs podName:1c2f1e93-f19e-4c0f-944e-7f52261f6e81 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:13.149754015 +0000 UTC m=+5.023485153 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-metrics-certs") pod "network-metrics-daemon-przs7" (UID: "1c2f1e93-f19e-4c0f-944e-7f52261f6e81") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:11.250304 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:11.250257 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4gkr\" (UniqueName: \"kubernetes.io/projected/b3f55f18-4143-4806-b9e4-989e153f7e38-kube-api-access-m4gkr\") pod \"network-check-target-c4b2t\" (UID: \"b3f55f18-4143-4806-b9e4-989e153f7e38\") " pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:11.250481 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:11.250435 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:42:11.250481 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:11.250460 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:42:11.250481 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:11.250474 2573 projected.go:196] Error preparing data for projected volume kube-api-access-m4gkr for pod openshift-network-diagnostics/network-check-target-c4b2t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:11.250660 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:11.250538 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3f55f18-4143-4806-b9e4-989e153f7e38-kube-api-access-m4gkr podName:b3f55f18-4143-4806-b9e4-989e153f7e38 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:13.250516139 +0000 UTC m=+5.124247273 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4gkr" (UniqueName: "kubernetes.io/projected/b3f55f18-4143-4806-b9e4-989e153f7e38-kube-api-access-m4gkr") pod "network-check-target-c4b2t" (UID: "b3f55f18-4143-4806-b9e4-989e153f7e38") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:11.683257 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:11.683161 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:37:10 +0000 UTC" deadline="2028-04-09 14:16:23.810265534 +0000 UTC" Oct 30 00:42:11.683257 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:11.683205 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12661h34m12.127065232s" Oct 30 00:42:11.701367 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:11.701332 2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:42:11.746861 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:11.746829 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:11.747058 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:11.746979 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-przs7" podUID="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" Oct 30 00:42:11.747408 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:11.747384 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:11.747504 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:11.747484 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c4b2t" podUID="b3f55f18-4143-4806-b9e4-989e153f7e38" Oct 30 00:42:11.747565 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:11.747549 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:11.747641 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:11.747618 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdkdr" podUID="b0661ca3-6f0a-4744-9d50-f2baa1a80b45" Oct 30 00:42:13.165733 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:13.164920 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-metrics-certs\") pod \"network-metrics-daemon-przs7\" (UID: \"1c2f1e93-f19e-4c0f-944e-7f52261f6e81\") " pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:13.165733 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:13.164994 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-original-pull-secret\") pod \"global-pull-secret-syncer-xdkdr\" (UID: \"b0661ca3-6f0a-4744-9d50-f2baa1a80b45\") " pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:13.165733 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:13.165134 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:13.165733 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:13.165203 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-original-pull-secret podName:b0661ca3-6f0a-4744-9d50-f2baa1a80b45 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:17.16518411 +0000 UTC m=+9.038915241 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-original-pull-secret") pod "global-pull-secret-syncer-xdkdr" (UID: "b0661ca3-6f0a-4744-9d50-f2baa1a80b45") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:13.165733 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:13.165273 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:13.165733 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:13.165307 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-metrics-certs podName:1c2f1e93-f19e-4c0f-944e-7f52261f6e81 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:17.165294669 +0000 UTC m=+9.039025805 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-metrics-certs") pod "network-metrics-daemon-przs7" (UID: "1c2f1e93-f19e-4c0f-944e-7f52261f6e81") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:13.265364 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:13.265327 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4gkr\" (UniqueName: \"kubernetes.io/projected/b3f55f18-4143-4806-b9e4-989e153f7e38-kube-api-access-m4gkr\") pod \"network-check-target-c4b2t\" (UID: \"b3f55f18-4143-4806-b9e4-989e153f7e38\") " pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:13.265531 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:13.265515 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:42:13.265611 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:13.265536 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:42:13.265611 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:13.265549 2573 projected.go:196] Error preparing data for projected volume kube-api-access-m4gkr for pod openshift-network-diagnostics/network-check-target-c4b2t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:13.265611 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:13.265605 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3f55f18-4143-4806-b9e4-989e153f7e38-kube-api-access-m4gkr podName:b3f55f18-4143-4806-b9e4-989e153f7e38 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:17.26558739 +0000 UTC m=+9.139318536 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4gkr" (UniqueName: "kubernetes.io/projected/b3f55f18-4143-4806-b9e4-989e153f7e38-kube-api-access-m4gkr") pod "network-check-target-c4b2t" (UID: "b3f55f18-4143-4806-b9e4-989e153f7e38") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:13.747470 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:13.746798 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:13.747470 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:13.747105 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-przs7" podUID="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" Oct 30 00:42:13.747470 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:13.747181 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:13.747470 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:13.747232 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c4b2t" podUID="b3f55f18-4143-4806-b9e4-989e153f7e38" Oct 30 00:42:13.747470 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:13.747287 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:13.747470 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:13.747358 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdkdr" podUID="b0661ca3-6f0a-4744-9d50-f2baa1a80b45" Oct 30 00:42:13.948151 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:13.948112 2573 generic.go:356] "Generic (PLEG): container finished" podID="49f04f3b-49d5-4c22-88a2-e33e106853ce" containerID="4a8d8c40faa74b5df1b39fedd3aeb654eb32844e06d0f7f1586a220b85b243a6" exitCode=0 Oct 30 00:42:13.948324 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:13.948175 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rhzn" event={"ID":"49f04f3b-49d5-4c22-88a2-e33e106853ce","Type":"ContainerDied","Data":"4a8d8c40faa74b5df1b39fedd3aeb654eb32844e06d0f7f1586a220b85b243a6"} Oct 30 00:42:14.956379 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:14.956335 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-89gxj" event={"ID":"271e26db-6a8b-4e46-afb7-139cbc57c1e1","Type":"ContainerStarted","Data":"6f403dd8261d2f143794a71c8b72e3ca793169ea21af73b45cffd97da80c78c9"} Oct 30 00:42:14.963998 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:14.963964 2573 generic.go:356] "Generic (PLEG): container finished" podID="91d0355fca0e17295f697c41968d06d7" containerID="ffbf7c1727a78c296c0d9d7895a47b4281c10c91811181dee30c24386b8d574c" exitCode=0 Oct 30 00:42:14.964145 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:14.964052 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal" event={"ID":"91d0355fca0e17295f697c41968d06d7","Type":"ContainerDied","Data":"ffbf7c1727a78c296c0d9d7895a47b4281c10c91811181dee30c24386b8d574c"} Oct 30 00:42:14.969861 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:14.969814 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-206.ec2.internal" event={"ID":"3b175f0644888f98ec9cdc35355eb1b8","Type":"ContainerStarted","Data":"f8616b8e19b9a4b4730f010b7c329b9922ac4e984ca6cbd319cc996f7bf50aac"} Oct 30 00:42:15.001507 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:15.001451 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-89gxj" podStartSLOduration=1.610684982 podStartE2EDuration="6.001436171s" totalImagesPullingTime="4.390751189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:15.00089491 +0000 UTC m=+6.874626066" watchObservedRunningTime="2026-10-30 00:42:15.001436171 +0000 UTC m=+6.875167326" Oct 30 00:42:15.044953 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:15.044889 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-206.ec2.internal" podStartSLOduration=5.044873023 podStartE2EDuration="5.044873023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:15.043868482 +0000 UTC m=+6.917599635" watchObservedRunningTime="2026-10-30 00:42:15.044873023 +0000 UTC m=+6.918604182" Oct 30 00:42:15.309397 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:15.309362 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-89gxj" Oct 30 00:42:15.310407 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:15.310379 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-89gxj" Oct 30 00:42:15.747189 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:15.747152 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:15.747424 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:15.747152 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:15.747424 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:15.747309 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:15.747424 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:15.747313 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-przs7" podUID="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" Oct 30 00:42:15.747424 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:15.747378 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c4b2t" podUID="b3f55f18-4143-4806-b9e4-989e153f7e38" Oct 30 00:42:15.747704 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:15.747444 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdkdr" podUID="b0661ca3-6f0a-4744-9d50-f2baa1a80b45" Oct 30 00:42:15.978461 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:15.978419 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g9ftg" event={"ID":"5dff88d3-2e35-40e9-8aea-208519485adc","Type":"ContainerStarted","Data":"34c8e6b836f153eeeddbdcc9be9095a74329781b35550d029bcd2abe21852b78"} Oct 30 00:42:15.982920 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:15.982522 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal" event={"ID":"91d0355fca0e17295f697c41968d06d7","Type":"ContainerStarted","Data":"b71f0b322f1e03a3d9d5b571df56131b5a19457428f9925e7b4093256229576e"} Oct 30 00:42:16.021327 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:16.021276 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-g9ftg" podStartSLOduration=1.478578085 podStartE2EDuration="7.021262518s" totalImagesPullingTime="5.542684433s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:16.020938201 +0000 UTC m=+7.894669357" watchObservedRunningTime="2026-10-30 00:42:16.021262518 +0000 UTC m=+7.894993672" Oct 30 00:42:16.985718 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:16.985688 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:42:17.204123 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:17.204082 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-original-pull-secret\") pod \"global-pull-secret-syncer-xdkdr\" (UID: \"b0661ca3-6f0a-4744-9d50-f2baa1a80b45\") " pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:17.204317 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:17.204183 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-metrics-certs\") pod \"network-metrics-daemon-przs7\" (UID: \"1c2f1e93-f19e-4c0f-944e-7f52261f6e81\") " pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:17.204392 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:17.204324 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:17.204447 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:17.204394 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-metrics-certs podName:1c2f1e93-f19e-4c0f-944e-7f52261f6e81 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:25.204373748 +0000 UTC m=+17.078104882 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-metrics-certs") pod "network-metrics-daemon-przs7" (UID: "1c2f1e93-f19e-4c0f-944e-7f52261f6e81") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:17.204820 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:17.204797 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:17.204929 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:17.204872 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-original-pull-secret podName:b0661ca3-6f0a-4744-9d50-f2baa1a80b45 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:25.20485519 +0000 UTC m=+17.078586337 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-original-pull-secret") pod "global-pull-secret-syncer-xdkdr" (UID: "b0661ca3-6f0a-4744-9d50-f2baa1a80b45") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:17.305205 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:17.305128 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4gkr\" (UniqueName: \"kubernetes.io/projected/b3f55f18-4143-4806-b9e4-989e153f7e38-kube-api-access-m4gkr\") pod \"network-check-target-c4b2t\" (UID: \"b3f55f18-4143-4806-b9e4-989e153f7e38\") " pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:17.305409 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:17.305383 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:42:17.305499 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:17.305418 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:42:17.305499 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:17.305434 2573 projected.go:196] Error preparing data for projected volume kube-api-access-m4gkr for pod openshift-network-diagnostics/network-check-target-c4b2t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:17.305610 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:17.305499 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3f55f18-4143-4806-b9e4-989e153f7e38-kube-api-access-m4gkr podName:b3f55f18-4143-4806-b9e4-989e153f7e38 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:25.305476727 +0000 UTC m=+17.179207874 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4gkr" (UniqueName: "kubernetes.io/projected/b3f55f18-4143-4806-b9e4-989e153f7e38-kube-api-access-m4gkr") pod "network-check-target-c4b2t" (UID: "b3f55f18-4143-4806-b9e4-989e153f7e38") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:17.746848 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:17.746813 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:17.747017 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:17.746986 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-przs7" podUID="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" Oct 30 00:42:17.747399 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:17.747370 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:17.747518 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:17.747484 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c4b2t" podUID="b3f55f18-4143-4806-b9e4-989e153f7e38" Oct 30 00:42:17.747604 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:17.747537 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:17.747604 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:17.747596 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdkdr" podUID="b0661ca3-6f0a-4744-9d50-f2baa1a80b45" Oct 30 00:42:18.992246 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:18.991833 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xmr6l" event={"ID":"da571014-afa3-4a40-a083-dc708bf83ba3","Type":"ContainerStarted","Data":"95caf615484160a426664374982f8a6357d2724a34fdd197de7c96733d754be3"} Oct 30 00:42:18.993308 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:18.993276 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6mjml" event={"ID":"995884d0-cb39-4c07-9dbe-5cd3529d5908","Type":"ContainerStarted","Data":"a58a9ee851303e864d6998a9a22a2110b8d6a6cb330ca7f4b007ef1b28881bec"} Oct 30 00:42:18.995828 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:18.995323 2573 generic.go:356] "Generic (PLEG): container finished" podID="49f04f3b-49d5-4c22-88a2-e33e106853ce" containerID="2ddecffc4c5270c2aec387819c4e23016a87a2db107be8f7a9379739e339b1ac" exitCode=0 Oct 30 00:42:18.995828 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:18.995360 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rhzn" event={"ID":"49f04f3b-49d5-4c22-88a2-e33e106853ce","Type":"ContainerDied","Data":"2ddecffc4c5270c2aec387819c4e23016a87a2db107be8f7a9379739e339b1ac"} Oct 30 00:42:19.034952 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:19.034890 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-xmr6l" podStartSLOduration=2.047006976 podStartE2EDuration="10.034872964s" totalImagesPullingTime="7.987865988s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:19.034642849 +0000 UTC m=+10.908374000" watchObservedRunningTime="2026-10-30 00:42:19.034872964 +0000 UTC m=+10.908604141" Oct 30 00:42:19.035095 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:19.035055 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal" podStartSLOduration=9.035046506 podStartE2EDuration="9.035046506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:16.064179706 +0000 UTC m=+7.937910861" watchObservedRunningTime="2026-10-30 00:42:19.035046506 +0000 UTC m=+10.908777658" Oct 30 00:42:19.075672 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:19.075617 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-6mjml" podStartSLOduration=1.8228385889999998 podStartE2EDuration="10.075598285s" totalImagesPullingTime="8.252759696s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:19.075354239 +0000 UTC m=+10.949085391" watchObservedRunningTime="2026-10-30 00:42:19.075598285 +0000 UTC m=+10.949329436" Oct 30 00:42:19.746586 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:19.746543 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:19.746674 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:19.746632 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:19.746674 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:19.746662 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:19.746827 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:19.746803 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c4b2t" podUID="b3f55f18-4143-4806-b9e4-989e153f7e38" Oct 30 00:42:19.746940 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:19.746918 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-przs7" podUID="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" Oct 30 00:42:19.747022 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:19.747002 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdkdr" podUID="b0661ca3-6f0a-4744-9d50-f2baa1a80b45" Oct 30 00:42:19.999275 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:19.999241 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xpngc" event={"ID":"7127ca6b-3962-47a9-9089-07bcba277bc7","Type":"ContainerStarted","Data":"b00879efd6ca47296c026e46c71eeb4a039b43d325f9572ab52fb568afbf0880"} Oct 30 00:42:20.002312 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:20.002270 2573 generic.go:356] "Generic (PLEG): container finished" podID="49f04f3b-49d5-4c22-88a2-e33e106853ce" containerID="0f9090f6f8f34f5af42dd21d7ec7a0422df95e56b90b425e1bb6960fd54ca22e" exitCode=0 Oct 30 00:42:20.003724 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:20.002903 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rhzn" event={"ID":"49f04f3b-49d5-4c22-88a2-e33e106853ce","Type":"ContainerDied","Data":"0f9090f6f8f34f5af42dd21d7ec7a0422df95e56b90b425e1bb6960fd54ca22e"} Oct 30 00:42:20.038505 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:20.038460 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xpngc" podStartSLOduration=2.6327730320000002 podStartE2EDuration="11.038444856s" totalImagesPullingTime="8.405671824s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:20.037815699 +0000 UTC m=+11.911546857" watchObservedRunningTime="2026-10-30 00:42:20.038444856 +0000 UTC m=+11.912176007" Oct 30 00:42:21.006487 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:21.006233 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" event={"ID":"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0","Type":"ContainerStarted","Data":"02e8ae2c3877111daa921653a4a8c2d2adfd80a69e6bead9e441d725b2cc9cb5"} Oct 30 00:42:21.008963 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:21.008926 2573 generic.go:356] "Generic (PLEG): container finished" podID="49f04f3b-49d5-4c22-88a2-e33e106853ce" containerID="cfd77bb0128b9d21dac7a3b7f566b2576c0faecedada1216db4e1855f4a04e10" exitCode=0 Oct 30 00:42:21.009090 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:21.009011 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rhzn" event={"ID":"49f04f3b-49d5-4c22-88a2-e33e106853ce","Type":"ContainerDied","Data":"cfd77bb0128b9d21dac7a3b7f566b2576c0faecedada1216db4e1855f4a04e10"} Oct 30 00:42:21.046437 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:21.046386 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" podStartSLOduration=1.906563345 podStartE2EDuration="12.04637002s" totalImagesPullingTime="10.139806675s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:21.045696357 +0000 UTC m=+12.919427521" watchObservedRunningTime="2026-10-30 00:42:21.04637002 +0000 UTC m=+12.920101170" Oct 30 00:42:21.746525 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:21.746492 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:21.746702 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:21.746586 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:21.746702 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:21.746606 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:21.746829 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:21.746703 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c4b2t" podUID="b3f55f18-4143-4806-b9e4-989e153f7e38" Oct 30 00:42:21.746880 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:21.746858 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-przs7" podUID="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" Oct 30 00:42:21.746939 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:21.746923 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdkdr" podUID="b0661ca3-6f0a-4744-9d50-f2baa1a80b45" Oct 30 00:42:23.018038 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:23.017965 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rhzn" event={"ID":"49f04f3b-49d5-4c22-88a2-e33e106853ce","Type":"ContainerStarted","Data":"59df6e494e5501738b2c9ce67ec9c0390dce9c1a45dcfeacd527ed40daf81ab7"} Oct 30 00:42:23.746505 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:23.746471 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:23.746663 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:23.746586 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c4b2t" podUID="b3f55f18-4143-4806-b9e4-989e153f7e38" Oct 30 00:42:23.746969 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:23.746949 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:23.747070 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:23.747049 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdkdr" podUID="b0661ca3-6f0a-4744-9d50-f2baa1a80b45" Oct 30 00:42:23.747139 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:23.747110 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:23.747253 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:23.747234 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-przs7" podUID="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" Oct 30 00:42:24.023676 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:24.023588 2573 generic.go:356] "Generic (PLEG): container finished" podID="49f04f3b-49d5-4c22-88a2-e33e106853ce" containerID="59df6e494e5501738b2c9ce67ec9c0390dce9c1a45dcfeacd527ed40daf81ab7" exitCode=0 Oct 30 00:42:24.023676 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:24.023649 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rhzn" event={"ID":"49f04f3b-49d5-4c22-88a2-e33e106853ce","Type":"ContainerDied","Data":"59df6e494e5501738b2c9ce67ec9c0390dce9c1a45dcfeacd527ed40daf81ab7"} Oct 30 00:42:25.028306 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:25.028138 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" event={"ID":"994c018e-e6c5-4d21-adac-9540676a67c0","Type":"ContainerStarted","Data":"88e19d079db2225c8349f589081eb7180ae501402640c566bc1d1c30cc4ace2a"} Oct 30 00:42:25.028630 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:25.028348 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" event={"ID":"994c018e-e6c5-4d21-adac-9540676a67c0","Type":"ContainerStarted","Data":"5b0f222e8327b4e6f7caf508df66953fbcb3d3b6c6cb6a1749a8460bee4b1313"} Oct 30 00:42:25.029684 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:25.029663 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" event={"ID":"1f1c6441-4ebb-48d6-9e09-bf69e431eb57","Type":"ContainerStarted","Data":"883bdab01abd70a701c7e31dee1740471f20a309c4b53628aeee03dced8f69c1"} Oct 30 00:42:25.032754 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:25.032734 2573 generic.go:356] "Generic (PLEG): container finished" podID="49f04f3b-49d5-4c22-88a2-e33e106853ce" containerID="0972806bff466c119f69edd4bcf0fda3ccfab64fd1598916981aa71a67afd145" exitCode=0 Oct 30 00:42:25.032860 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:25.032777 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rhzn" event={"ID":"49f04f3b-49d5-4c22-88a2-e33e106853ce","Type":"ContainerDied","Data":"0972806bff466c119f69edd4bcf0fda3ccfab64fd1598916981aa71a67afd145"} Oct 30 00:42:25.264402 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:25.264197 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-original-pull-secret\") pod \"global-pull-secret-syncer-xdkdr\" (UID: \"b0661ca3-6f0a-4744-9d50-f2baa1a80b45\") " pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:25.264565 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:25.264455 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-metrics-certs\") pod \"network-metrics-daemon-przs7\" (UID: \"1c2f1e93-f19e-4c0f-944e-7f52261f6e81\") " pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:25.264565 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:25.264336 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:25.264565 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:25.264531 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-original-pull-secret podName:b0661ca3-6f0a-4744-9d50-f2baa1a80b45 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:41.264511055 +0000 UTC m=+33.138242198 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-original-pull-secret") pod "global-pull-secret-syncer-xdkdr" (UID: "b0661ca3-6f0a-4744-9d50-f2baa1a80b45") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:25.264565 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:25.264565 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:25.264700 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:25.264596 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-metrics-certs podName:1c2f1e93-f19e-4c0f-944e-7f52261f6e81 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:41.264588877 +0000 UTC m=+33.138320006 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-metrics-certs") pod "network-metrics-daemon-przs7" (UID: "1c2f1e93-f19e-4c0f-944e-7f52261f6e81") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:25.365099 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:25.365033 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4gkr\" (UniqueName: \"kubernetes.io/projected/b3f55f18-4143-4806-b9e4-989e153f7e38-kube-api-access-m4gkr\") pod \"network-check-target-c4b2t\" (UID: \"b3f55f18-4143-4806-b9e4-989e153f7e38\") " pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:25.365335 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:25.365203 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:42:25.365335 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:25.365229 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:42:25.365335 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:25.365244 2573 projected.go:196] Error preparing data for projected volume kube-api-access-m4gkr for pod openshift-network-diagnostics/network-check-target-c4b2t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:25.365335 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:25.365302 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3f55f18-4143-4806-b9e4-989e153f7e38-kube-api-access-m4gkr podName:b3f55f18-4143-4806-b9e4-989e153f7e38 nodeName:}" failed. No retries permitted until 2026-10-30 00:42:41.365285145 +0000 UTC m=+33.239016289 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4gkr" (UniqueName: "kubernetes.io/projected/b3f55f18-4143-4806-b9e4-989e153f7e38-kube-api-access-m4gkr") pod "network-check-target-c4b2t" (UID: "b3f55f18-4143-4806-b9e4-989e153f7e38") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:25.746691 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:25.746669 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:25.746855 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:25.746699 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:25.746855 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:25.746791 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-przs7" podUID="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" Oct 30 00:42:25.746855 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:25.746830 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:25.746958 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:25.746900 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c4b2t" podUID="b3f55f18-4143-4806-b9e4-989e153f7e38" Oct 30 00:42:25.746994 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:25.746957 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdkdr" podUID="b0661ca3-6f0a-4744-9d50-f2baa1a80b45" Oct 30 00:42:25.847368 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:25.847343 2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 00:42:26.038161 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:26.038083 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" event={"ID":"994c018e-e6c5-4d21-adac-9540676a67c0","Type":"ContainerStarted","Data":"599b2b22667408efef5e1800f2c5a2809d8aed6dc9e4178827bf3405f2fbdf12"} Oct 30 00:42:26.038678 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:26.038192 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" event={"ID":"994c018e-e6c5-4d21-adac-9540676a67c0","Type":"ContainerStarted","Data":"c6c6b3981ac90f835cfeb56e07ba3d82c69a96db532d00de53e89e178219a46d"} Oct 30 00:42:26.038678 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:26.038227 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" event={"ID":"994c018e-e6c5-4d21-adac-9540676a67c0","Type":"ContainerStarted","Data":"5a1fb0bab9ebf155900e117be803c191ec1fd81f9e223798b28a413f2a754290"} Oct 30 00:42:26.038678 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:26.038255 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" event={"ID":"994c018e-e6c5-4d21-adac-9540676a67c0","Type":"ContainerStarted","Data":"3541f3a0ad92e9757144e08a3a77bfd61970943547e9184aa586efbfd75fcd5f"} Oct 30 00:42:26.039819 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:26.039790 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" event={"ID":"1f1c6441-4ebb-48d6-9e09-bf69e431eb57","Type":"ContainerStarted","Data":"2497110ae7a18853bc1cd223363757c7f7f945f43af8d92948706aba344342d7"} Oct 30 00:42:26.042857 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:26.042833 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rhzn" event={"ID":"49f04f3b-49d5-4c22-88a2-e33e106853ce","Type":"ContainerStarted","Data":"18a58a2eabee8dca7a406dbb12a838c1ef1c1b006380e5cea72702efc90f865d"} Oct 30 00:42:26.087721 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:26.087679 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5rhzn" podStartSLOduration=7.259431653 podStartE2EDuration="17.087667408s" totalImagesPullingTime="9.828235755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:42:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:26.086895506 +0000 UTC m=+17.960626657" watchObservedRunningTime="2026-10-30 00:42:26.087667408 +0000 UTC m=+17.961398558" Oct 30 00:42:26.703703 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:26.703606 2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:42:25.847363126Z","UUID":"3dabe1c6-57c3-4202-ac0e-330c274417b3","Handler":null,"Name":"","Endpoint":""} Oct 30 00:42:26.705222 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:26.705202 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 00:42:26.705312 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:26.705229 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 00:42:27.046406 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:27.046370 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" event={"ID":"1f1c6441-4ebb-48d6-9e09-bf69e431eb57","Type":"ContainerStarted","Data":"b7b3ea1b4930d3a6e1ba04c44da5a6da6e80c6f79a2176ff48f4cea24df8ebed"} Oct 30 00:42:27.088738 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:27.088692 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" podStartSLOduration=1.468214714 podStartE2EDuration="18.088680117s" totalImagesPullingTime="16.620465403s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:42:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=3 observedRunningTime="2026-10-30 00:42:27.088318919 +0000 UTC m=+18.962050069" watchObservedRunningTime="2026-10-30 00:42:27.088680117 +0000 UTC m=+18.962411268" Oct 30 00:42:27.746666 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:27.746633 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:27.746666 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:27.746670 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:27.746897 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:27.746666 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:27.746897 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:27.746738 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdkdr" podUID="b0661ca3-6f0a-4744-9d50-f2baa1a80b45" Oct 30 00:42:27.746897 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:27.746833 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-przs7" podUID="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" Oct 30 00:42:27.747049 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:27.746905 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c4b2t" podUID="b3f55f18-4143-4806-b9e4-989e153f7e38" Oct 30 00:42:28.051500 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:28.051418 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" event={"ID":"994c018e-e6c5-4d21-adac-9540676a67c0","Type":"ContainerStarted","Data":"73fac0cfc0794262d605bb82dce37cfeb246c1266d31023b58595e3d5b8fe066"} Oct 30 00:42:29.746870 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:29.746685 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:29.747363 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:29.746731 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:29.747363 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:29.746739 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:29.747363 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:29.747025 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdkdr" podUID="b0661ca3-6f0a-4744-9d50-f2baa1a80b45" Oct 30 00:42:29.747363 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:29.746938 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c4b2t" podUID="b3f55f18-4143-4806-b9e4-989e153f7e38" Oct 30 00:42:29.747363 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:29.747093 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-przs7" podUID="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" Oct 30 00:42:30.058861 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:30.058789 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" event={"ID":"994c018e-e6c5-4d21-adac-9540676a67c0","Type":"ContainerStarted","Data":"1159e30a231c71398affe097824367bce1de6bc28125a412718b34687c5cbcf7"} Oct 30 00:42:30.059093 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:30.059067 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:30.059224 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:30.059140 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:30.059224 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:30.059182 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:30.074255 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:30.074229 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:30.074700 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:30.074624 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:42:30.148267 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:30.148222 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" podStartSLOduration=6.165868552 podStartE2EDuration="21.14821023s" totalImagesPullingTime="14.982341678s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:30.147494454 +0000 UTC m=+22.021225609" watchObservedRunningTime="2026-10-30 00:42:30.14821023 +0000 UTC m=+22.021941381" Oct 30 00:42:30.524305 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:30.524263 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:30.654332 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:30.654288 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:31.425334 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:31.425264 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-przs7"] Oct 30 00:42:31.425784 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:31.425397 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:31.425784 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:31.425502 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-przs7" podUID="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" Oct 30 00:42:31.439066 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:31.439040 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xdkdr"] Oct 30 00:42:31.439206 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:31.439160 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:31.439249 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:31.439236 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdkdr" podUID="b0661ca3-6f0a-4744-9d50-f2baa1a80b45" Oct 30 00:42:31.439514 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:31.439494 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c4b2t"] Oct 30 00:42:31.439591 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:31.439581 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:31.439662 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:31.439648 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c4b2t" podUID="b3f55f18-4143-4806-b9e4-989e153f7e38" Oct 30 00:42:32.745299 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:32.745266 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:32.746255 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:32.746223 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:32.746377 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:32.746317 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdkdr" podUID="b0661ca3-6f0a-4744-9d50-f2baa1a80b45" Oct 30 00:42:32.746415 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:32.746386 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:32.746480 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:32.746464 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c4b2t" podUID="b3f55f18-4143-4806-b9e4-989e153f7e38" Oct 30 00:42:33.747044 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:33.747018 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:33.747381 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:33.747117 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-przs7" podUID="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" Oct 30 00:42:34.308267 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:34.308237 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:34.390135 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:34.390099 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:34.475616 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:34.475589 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:34.747061 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:34.747036 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:34.747445 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:34.747134 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c4b2t" podUID="b3f55f18-4143-4806-b9e4-989e153f7e38" Oct 30 00:42:34.747445 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:34.747236 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:34.747445 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:34.747336 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdkdr" podUID="b0661ca3-6f0a-4744-9d50-f2baa1a80b45" Oct 30 00:42:35.017753 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:35.017678 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:35.117838 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:35.117797 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:35.218627 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:35.218600 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:35.731896 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:35.731865 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:35.747002 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:35.746982 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:35.747099 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:35.747083 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-przs7" podUID="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" Oct 30 00:42:35.844119 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:35.844043 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:36.226435 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:36.226400 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:36.746368 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:36.746338 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:36.746527 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:36.746375 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:36.746527 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:36.746442 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c4b2t" podUID="b3f55f18-4143-4806-b9e4-989e153f7e38" Oct 30 00:42:36.746619 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:36.746550 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdkdr" podUID="b0661ca3-6f0a-4744-9d50-f2baa1a80b45" Oct 30 00:42:36.829456 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:36.829432 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6mjml_995884d0-cb39-4c07-9dbe-5cd3529d5908/dns-node-resolver/0.log" Oct 30 00:42:37.231109 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:37.230943 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:37.333156 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:37.333086 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:37.431741 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:37.431689 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:37.544773 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:37.544727 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:37.623965 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:37.623888 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:37.723069 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:37.723038 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:37.746987 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:37.746968 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:37.747104 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:37.747086 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-przs7" podUID="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" Oct 30 00:42:37.829492 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:37.829472 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g9ftg_5dff88d3-2e35-40e9-8aea-208519485adc/node-ca/0.log" Oct 30 00:42:37.868821 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:37.868797 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-89gxj" Oct 30 00:42:37.868934 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:37.868921 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:42:37.869398 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:37.869379 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-89gxj" Oct 30 00:42:37.899610 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:37.899540 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:38.074501 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:38.074475 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:38.747829 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:38.747792 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:38.747992 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:38.747953 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdkdr" podUID="b0661ca3-6f0a-4744-9d50-f2baa1a80b45" Oct 30 00:42:38.748049 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:38.748001 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:38.748112 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:38.748093 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c4b2t" podUID="b3f55f18-4143-4806-b9e4-989e153f7e38" Oct 30 00:42:38.844564 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:38.844529 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:39.096729 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:39.096661 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:39.300493 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:39.300461 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:39.503936 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:39.503904 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:39.710636 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:39.710608 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:39.746920 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:39.746893 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:39.747082 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:39.747018 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-przs7" podUID="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" Oct 30 00:42:39.896249 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:39.896178 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:40.747212 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:40.747185 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:40.747380 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:40.747277 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c4b2t" podUID="b3f55f18-4143-4806-b9e4-989e153f7e38" Oct 30 00:42:40.747380 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:40.747348 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:40.747459 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:40.747429 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdkdr" podUID="b0661ca3-6f0a-4744-9d50-f2baa1a80b45" Oct 30 00:42:41.089246 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:41.089171 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:41.278547 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:41.278523 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-metrics-certs\") pod \"network-metrics-daemon-przs7\" (UID: \"1c2f1e93-f19e-4c0f-944e-7f52261f6e81\") " pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:41.278784 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:41.278559 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-original-pull-secret\") pod \"global-pull-secret-syncer-xdkdr\" (UID: \"b0661ca3-6f0a-4744-9d50-f2baa1a80b45\") " pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:41.278784 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:41.278657 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:41.278784 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:41.278664 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:41.278784 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:41.278698 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-original-pull-secret podName:b0661ca3-6f0a-4744-9d50-f2baa1a80b45 nodeName:}" failed. No retries permitted until 2026-10-30 00:43:13.278687317 +0000 UTC m=+65.152418447 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-original-pull-secret") pod "global-pull-secret-syncer-xdkdr" (UID: "b0661ca3-6f0a-4744-9d50-f2baa1a80b45") : object "kube-system"/"original-pull-secret" not registered Oct 30 00:42:41.278784 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:41.278711 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-metrics-certs podName:1c2f1e93-f19e-4c0f-944e-7f52261f6e81 nodeName:}" failed. No retries permitted until 2026-10-30 00:43:13.278704769 +0000 UTC m=+65.152435897 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-metrics-certs") pod "network-metrics-daemon-przs7" (UID: "1c2f1e93-f19e-4c0f-944e-7f52261f6e81") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 00:42:41.288896 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:41.288874 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:41.379008 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:41.378953 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4gkr\" (UniqueName: \"kubernetes.io/projected/b3f55f18-4143-4806-b9e4-989e153f7e38-kube-api-access-m4gkr\") pod \"network-check-target-c4b2t\" (UID: \"b3f55f18-4143-4806-b9e4-989e153f7e38\") " pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:41.379093 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:41.379086 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 00:42:41.379133 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:41.379102 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 00:42:41.379133 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:41.379110 2573 projected.go:196] Error preparing data for projected volume kube-api-access-m4gkr for pod openshift-network-diagnostics/network-check-target-c4b2t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:41.379198 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:41.379149 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3f55f18-4143-4806-b9e4-989e153f7e38-kube-api-access-m4gkr podName:b3f55f18-4143-4806-b9e4-989e153f7e38 nodeName:}" failed. No retries permitted until 2026-10-30 00:43:13.379137728 +0000 UTC m=+65.252868857 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4gkr" (UniqueName: "kubernetes.io/projected/b3f55f18-4143-4806-b9e4-989e153f7e38-kube-api-access-m4gkr") pod "network-check-target-c4b2t" (UID: "b3f55f18-4143-4806-b9e4-989e153f7e38") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 00:42:41.489090 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:41.489055 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:41.697558 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:41.697477 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:41.746914 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:41.746885 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:41.747064 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:41.746983 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-przs7" podUID="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" Oct 30 00:42:41.888668 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:41.888641 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:41.988743 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:41.988712 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:42.147290 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:42.147262 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:42.746976 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:42.746939 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:42.746976 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:42.746967 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:42.747217 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:42.747043 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdkdr" podUID="b0661ca3-6f0a-4744-9d50-f2baa1a80b45" Oct 30 00:42:42.747217 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:42.747170 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c4b2t" podUID="b3f55f18-4143-4806-b9e4-989e153f7e38" Oct 30 00:42:42.886898 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:42.886853 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:43.287856 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:43.287818 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:43.487676 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:43.487646 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:43.578657 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:43.578584 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:43.655920 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:43.655855 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:43.746629 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:43.746595 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:43.746849 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:43.746737 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-przs7" podUID="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" Oct 30 00:42:43.755616 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:43.755585 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:43.989126 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:43.989100 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:44.093492 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:44.093466 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:44.747020 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:44.746982 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:44.747510 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:44.747000 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:44.747510 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:44.747087 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c4b2t" podUID="b3f55f18-4143-4806-b9e4-989e153f7e38" Oct 30 00:42:44.747510 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:44.747161 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdkdr" podUID="b0661ca3-6f0a-4744-9d50-f2baa1a80b45" Oct 30 00:42:44.887114 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:44.887078 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:45.286861 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:45.286829 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:45.490134 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:45.490106 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:45.694257 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:45.694184 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:45.746400 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:45.746376 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:45.746514 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:45.746478 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-przs7" podUID="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" Oct 30 00:42:46.092050 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:46.092020 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:46.471462 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:46.471436 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:46.571319 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:46.571295 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:46.747203 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:46.747134 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:46.747335 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:46.747137 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:46.747335 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:46.747220 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c4b2t" podUID="b3f55f18-4143-4806-b9e4-989e153f7e38" Oct 30 00:42:46.747335 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:46.747291 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdkdr" podUID="b0661ca3-6f0a-4744-9d50-f2baa1a80b45" Oct 30 00:42:46.891831 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:46.891801 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:47.311497 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:47.311468 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:47.488551 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:47.488526 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:47.685564 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:47.685496 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:47.747198 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:47.747176 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:47.747354 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:47.747295 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-przs7" podUID="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" Oct 30 00:42:48.088907 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:48.088874 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:48.287721 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:48.287696 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:48.747499 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:48.747470 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:48.747891 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:48.747552 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdkdr" podUID="b0661ca3-6f0a-4744-9d50-f2baa1a80b45" Oct 30 00:42:48.747891 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:48.747631 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:48.747891 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:48.747701 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c4b2t" podUID="b3f55f18-4143-4806-b9e4-989e153f7e38" Oct 30 00:42:49.746322 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:49.746293 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:49.746479 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:49.746393 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-przs7" podUID="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" Oct 30 00:42:50.749220 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:50.749187 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:50.749625 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:50.749297 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c4b2t" podUID="b3f55f18-4143-4806-b9e4-989e153f7e38" Oct 30 00:42:50.749625 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:50.749361 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:50.749625 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:50.749440 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdkdr" podUID="b0661ca3-6f0a-4744-9d50-f2baa1a80b45" Oct 30 00:42:51.126169 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:51.126090 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:51.220251 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:51.220219 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:51.320156 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:51.320124 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:51.420439 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:51.420374 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:51.520027 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:51.519998 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:51.630590 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:51.630565 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:51.747035 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:51.747013 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:51.747155 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:51.747103 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-przs7" podUID="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" Oct 30 00:42:51.783252 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:51.783227 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:52.746330 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:52.746295 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:52.746498 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:52.746424 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c4b2t" podUID="b3f55f18-4143-4806-b9e4-989e153f7e38" Oct 30 00:42:52.746498 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:52.746460 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:52.746579 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:52.746534 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdkdr" podUID="b0661ca3-6f0a-4744-9d50-f2baa1a80b45" Oct 30 00:42:53.746948 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:53.746919 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:53.747323 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:53.747018 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-przs7" podUID="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" Oct 30 00:42:54.322373 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:54.322344 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:54.396299 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:54.396273 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:54.749029 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:54.749003 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:54.749400 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:54.749007 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:54.749400 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:54.749163 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdkdr" podUID="b0661ca3-6f0a-4744-9d50-f2baa1a80b45" Oct 30 00:42:54.749400 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:54.749087 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c4b2t" podUID="b3f55f18-4143-4806-b9e4-989e153f7e38" Oct 30 00:42:55.747071 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:55.747041 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:55.747240 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:55.747141 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-przs7" podUID="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" Oct 30 00:42:56.746556 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:56.746517 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:56.746954 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:56.746562 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:56.746954 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:56.746654 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c4b2t" podUID="b3f55f18-4143-4806-b9e4-989e153f7e38" Oct 30 00:42:56.746954 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:56.746751 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdkdr" podUID="b0661ca3-6f0a-4744-9d50-f2baa1a80b45" Oct 30 00:42:57.746972 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:57.746937 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:57.747364 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:57.747044 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-przs7" podUID="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" Oct 30 00:42:58.747234 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:58.747197 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:42:58.747682 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:58.747300 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c4b2t" podUID="b3f55f18-4143-4806-b9e4-989e153f7e38" Oct 30 00:42:58.747682 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:58.747379 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:42:58.747682 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:58.747481 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdkdr" podUID="b0661ca3-6f0a-4744-9d50-f2baa1a80b45" Oct 30 00:42:58.968519 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:58.968484 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:59.063024 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:59.062937 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:59.156269 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:59.156236 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:42:59.747134 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:59.747090 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:42:59.747320 ip-10-0-140-206 kubenswrapper[2573]: E1030 00:42:59.747236 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-przs7" podUID="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" Oct 30 00:42:59.898815 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:59.898779 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-206.ec2.internal" event="NodeReady" Oct 30 00:42:59.898991 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:59.898925 2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 00:42:59.954989 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:59.954960 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5jhd7"] Oct 30 00:42:59.957643 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:59.957622 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jpp4m"] Oct 30 00:42:59.957793 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:59.957751 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5jhd7" Oct 30 00:42:59.960217 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:59.960200 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jpp4m" Oct 30 00:42:59.974563 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:59.974540 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xmsbl"] Oct 30 00:42:59.976773 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:59.976737 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 00:42:59.977323 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:59.977307 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:42:59.977803 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:59.977771 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5jhd7"] Oct 30 00:42:59.979672 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:59.979639 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 00:42:59.980244 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:59.979682 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 00:42:59.980411 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:59.980383 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jpp4m"] Oct 30 00:42:59.980516 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:59.980499 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 00:42:59.987563 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:59.987544 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ph2zs\"" Oct 30 00:42:59.987853 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:59.987834 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5gz8r\"" Oct 30 00:42:59.987853 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:59.987849 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 00:42:59.993127 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:59.993111 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b5pp6\"" Oct 30 00:42:59.993315 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:42:59.993301 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 00:43:00.004643 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.004579 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xmsbl"] Oct 30 00:43:00.077452 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.077425 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 00:43:00.077452 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.077455 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 00:43:00.077624 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.077460 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 00:43:00.117251 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.117221 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/81258c01-d653-4a05-87ca-8373e7fad83d-config-volume\") pod \"dns-default-5jhd7\" (UID: \"81258c01-d653-4a05-87ca-8373e7fad83d\") " pod="openshift-dns/dns-default-5jhd7" Oct 30 00:43:00.117251 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.117253 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81258c01-d653-4a05-87ca-8373e7fad83d-metrics-tls\") pod \"dns-default-5jhd7\" (UID: \"81258c01-d653-4a05-87ca-8373e7fad83d\") " pod="openshift-dns/dns-default-5jhd7" Oct 30 00:43:00.117459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.117272 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9srn\" (UniqueName: \"kubernetes.io/projected/0b22701a-4358-4fd4-92d9-bd838f5af629-kube-api-access-b9srn\") pod \"insights-runtime-extractor-xmsbl\" (UID: \"0b22701a-4358-4fd4-92d9-bd838f5af629\") " pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:43:00.117459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.117330 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b74st\" (UniqueName: \"kubernetes.io/projected/81258c01-d653-4a05-87ca-8373e7fad83d-kube-api-access-b74st\") pod \"dns-default-5jhd7\" (UID: \"81258c01-d653-4a05-87ca-8373e7fad83d\") " pod="openshift-dns/dns-default-5jhd7" Oct 30 00:43:00.117459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.117413 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8kj9\" (UniqueName: \"kubernetes.io/projected/653fb098-e2a1-4674-9baa-6975e9dd0fec-kube-api-access-s8kj9\") pod \"ingress-canary-jpp4m\" (UID: \"653fb098-e2a1-4674-9baa-6975e9dd0fec\") " pod="openshift-ingress-canary/ingress-canary-jpp4m" Oct 30 00:43:00.117459 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.117448 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b22701a-4358-4fd4-92d9-bd838f5af629-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmsbl\" (UID: \"0b22701a-4358-4fd4-92d9-bd838f5af629\") " pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:43:00.117657 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.117467 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b22701a-4358-4fd4-92d9-bd838f5af629-crio-socket\") pod \"insights-runtime-extractor-xmsbl\" (UID: \"0b22701a-4358-4fd4-92d9-bd838f5af629\") " pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:43:00.117657 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.117484 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81258c01-d653-4a05-87ca-8373e7fad83d-tmp-dir\") pod \"dns-default-5jhd7\" (UID: \"81258c01-d653-4a05-87ca-8373e7fad83d\") " pod="openshift-dns/dns-default-5jhd7" Oct 30 00:43:00.117657 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.117557 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/653fb098-e2a1-4674-9baa-6975e9dd0fec-cert\") pod \"ingress-canary-jpp4m\" (UID: \"653fb098-e2a1-4674-9baa-6975e9dd0fec\") " pod="openshift-ingress-canary/ingress-canary-jpp4m" Oct 30 00:43:00.117657 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.117605 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b22701a-4358-4fd4-92d9-bd838f5af629-data-volume\") pod \"insights-runtime-extractor-xmsbl\" (UID: \"0b22701a-4358-4fd4-92d9-bd838f5af629\") " pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:43:00.117657 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.117641 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b22701a-4358-4fd4-92d9-bd838f5af629-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmsbl\" (UID: \"0b22701a-4358-4fd4-92d9-bd838f5af629\") " pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:43:00.218868 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.218835 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/81258c01-d653-4a05-87ca-8373e7fad83d-config-volume\") pod \"dns-default-5jhd7\" (UID: \"81258c01-d653-4a05-87ca-8373e7fad83d\") " pod="openshift-dns/dns-default-5jhd7" Oct 30 00:43:00.218868 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.218868 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81258c01-d653-4a05-87ca-8373e7fad83d-metrics-tls\") pod \"dns-default-5jhd7\" (UID: \"81258c01-d653-4a05-87ca-8373e7fad83d\") " pod="openshift-dns/dns-default-5jhd7" Oct 30 00:43:00.219098 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.218891 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9srn\" (UniqueName: \"kubernetes.io/projected/0b22701a-4358-4fd4-92d9-bd838f5af629-kube-api-access-b9srn\") pod \"insights-runtime-extractor-xmsbl\" (UID: \"0b22701a-4358-4fd4-92d9-bd838f5af629\") " pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:43:00.219098 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.218915 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b74st\" (UniqueName: \"kubernetes.io/projected/81258c01-d653-4a05-87ca-8373e7fad83d-kube-api-access-b74st\") pod \"dns-default-5jhd7\" (UID: \"81258c01-d653-4a05-87ca-8373e7fad83d\") " pod="openshift-dns/dns-default-5jhd7" Oct 30 00:43:00.219098 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.218948 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8kj9\" (UniqueName: \"kubernetes.io/projected/653fb098-e2a1-4674-9baa-6975e9dd0fec-kube-api-access-s8kj9\") pod \"ingress-canary-jpp4m\" (UID: \"653fb098-e2a1-4674-9baa-6975e9dd0fec\") " pod="openshift-ingress-canary/ingress-canary-jpp4m" Oct 30 00:43:00.219098 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.218989 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b22701a-4358-4fd4-92d9-bd838f5af629-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmsbl\" (UID: \"0b22701a-4358-4fd4-92d9-bd838f5af629\") " pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:43:00.219098 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.219013 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b22701a-4358-4fd4-92d9-bd838f5af629-crio-socket\") pod \"insights-runtime-extractor-xmsbl\" (UID: \"0b22701a-4358-4fd4-92d9-bd838f5af629\") " pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:43:00.219098 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.219036 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81258c01-d653-4a05-87ca-8373e7fad83d-tmp-dir\") pod \"dns-default-5jhd7\" (UID: \"81258c01-d653-4a05-87ca-8373e7fad83d\") " pod="openshift-dns/dns-default-5jhd7" Oct 30 00:43:00.219098 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.219056 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/653fb098-e2a1-4674-9baa-6975e9dd0fec-cert\") pod \"ingress-canary-jpp4m\" (UID: \"653fb098-e2a1-4674-9baa-6975e9dd0fec\") " pod="openshift-ingress-canary/ingress-canary-jpp4m" Oct 30 00:43:00.219098 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.219086 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b22701a-4358-4fd4-92d9-bd838f5af629-data-volume\") pod \"insights-runtime-extractor-xmsbl\" (UID: \"0b22701a-4358-4fd4-92d9-bd838f5af629\") " pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:43:00.219471 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.219106 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b22701a-4358-4fd4-92d9-bd838f5af629-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmsbl\" (UID: \"0b22701a-4358-4fd4-92d9-bd838f5af629\") " pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:43:00.219471 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.219121 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b22701a-4358-4fd4-92d9-bd838f5af629-crio-socket\") pod \"insights-runtime-extractor-xmsbl\" (UID: \"0b22701a-4358-4fd4-92d9-bd838f5af629\") " pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:43:00.219569 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.219504 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81258c01-d653-4a05-87ca-8373e7fad83d-tmp-dir\") pod \"dns-default-5jhd7\" (UID: \"81258c01-d653-4a05-87ca-8373e7fad83d\") " pod="openshift-dns/dns-default-5jhd7" Oct 30 00:43:00.219569 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.219538 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b22701a-4358-4fd4-92d9-bd838f5af629-data-volume\") pod \"insights-runtime-extractor-xmsbl\" (UID: \"0b22701a-4358-4fd4-92d9-bd838f5af629\") " pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:43:00.219814 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.219784 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b22701a-4358-4fd4-92d9-bd838f5af629-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmsbl\" (UID: \"0b22701a-4358-4fd4-92d9-bd838f5af629\") " pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:43:00.220057 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.220032 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/81258c01-d653-4a05-87ca-8373e7fad83d-config-volume\") pod \"dns-default-5jhd7\" (UID: \"81258c01-d653-4a05-87ca-8373e7fad83d\") " pod="openshift-dns/dns-default-5jhd7" Oct 30 00:43:00.222719 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.222695 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b22701a-4358-4fd4-92d9-bd838f5af629-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmsbl\" (UID: \"0b22701a-4358-4fd4-92d9-bd838f5af629\") " pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:43:00.222823 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.222784 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81258c01-d653-4a05-87ca-8373e7fad83d-metrics-tls\") pod \"dns-default-5jhd7\" (UID: \"81258c01-d653-4a05-87ca-8373e7fad83d\") " pod="openshift-dns/dns-default-5jhd7" Oct 30 00:43:00.222893 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.222871 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/653fb098-e2a1-4674-9baa-6975e9dd0fec-cert\") pod \"ingress-canary-jpp4m\" (UID: \"653fb098-e2a1-4674-9baa-6975e9dd0fec\") " pod="openshift-ingress-canary/ingress-canary-jpp4m" Oct 30 00:43:00.239448 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.239425 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b74st\" (UniqueName: \"kubernetes.io/projected/81258c01-d653-4a05-87ca-8373e7fad83d-kube-api-access-b74st\") pod \"dns-default-5jhd7\" (UID: \"81258c01-d653-4a05-87ca-8373e7fad83d\") " pod="openshift-dns/dns-default-5jhd7" Oct 30 00:43:00.239552 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.239459 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9srn\" (UniqueName: \"kubernetes.io/projected/0b22701a-4358-4fd4-92d9-bd838f5af629-kube-api-access-b9srn\") pod \"insights-runtime-extractor-xmsbl\" (UID: \"0b22701a-4358-4fd4-92d9-bd838f5af629\") " pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:43:00.239552 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.239542 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8kj9\" (UniqueName: \"kubernetes.io/projected/653fb098-e2a1-4674-9baa-6975e9dd0fec-kube-api-access-s8kj9\") pod \"ingress-canary-jpp4m\" (UID: \"653fb098-e2a1-4674-9baa-6975e9dd0fec\") " pod="openshift-ingress-canary/ingress-canary-jpp4m" Oct 30 00:43:00.272566 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.272499 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5jhd7" Oct 30 00:43:00.274580 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.274559 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jpp4m" Oct 30 00:43:00.294145 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.294113 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:43:00.585194 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.585000 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5jhd7"] Oct 30 00:43:00.587740 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:43:00.587715 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81258c01_d653_4a05_87ca_8373e7fad83d.slice/crio-e23232e117081451795faf1fc0b232b8ae80acdaa22d5d0a82c2d92146c78453 WatchSource:0}: Error finding container e23232e117081451795faf1fc0b232b8ae80acdaa22d5d0a82c2d92146c78453: Status 404 returned error can't find the container with id e23232e117081451795faf1fc0b232b8ae80acdaa22d5d0a82c2d92146c78453 Oct 30 00:43:00.597666 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.597640 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jpp4m"] Oct 30 00:43:00.600094 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:43:00.600067 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653fb098_e2a1_4674_9baa_6975e9dd0fec.slice/crio-874ddcd3a5bb7eb513a1939007b192044d8483cb5e1d5b6603821d7adfc204e6 WatchSource:0}: Error finding container 874ddcd3a5bb7eb513a1939007b192044d8483cb5e1d5b6603821d7adfc204e6: Status 404 returned error can't find the container with id 874ddcd3a5bb7eb513a1939007b192044d8483cb5e1d5b6603821d7adfc204e6 Oct 30 00:43:00.601833 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.601810 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xmsbl"] Oct 30 00:43:00.604895 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:43:00.604875 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b22701a_4358_4fd4_92d9_bd838f5af629.slice/crio-01c112ff76552443b9e39c5b79208ec96ef2aa7f3e6b7573a69ed107b9f71355 WatchSource:0}: Error finding container 01c112ff76552443b9e39c5b79208ec96ef2aa7f3e6b7573a69ed107b9f71355: Status 404 returned error can't find the container with id 01c112ff76552443b9e39c5b79208ec96ef2aa7f3e6b7573a69ed107b9f71355 Oct 30 00:43:00.711422 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.711380 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:00.747227 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.747192 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:43:00.747227 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.747227 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:43:00.763733 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.763707 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 00:43:00.763733 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.763718 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 00:43:00.763890 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.763756 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 00:43:00.763890 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.763806 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vdsv4\"" Oct 30 00:43:00.763956 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.763903 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-p4p9c\"" Oct 30 00:43:00.814094 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:00.814066 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:01.123319 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:01.123280 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5jhd7" event={"ID":"81258c01-d653-4a05-87ca-8373e7fad83d","Type":"ContainerStarted","Data":"e23232e117081451795faf1fc0b232b8ae80acdaa22d5d0a82c2d92146c78453"} Oct 30 00:43:01.125133 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:01.125032 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xmsbl" event={"ID":"0b22701a-4358-4fd4-92d9-bd838f5af629","Type":"ContainerStarted","Data":"86aee778056186b65d3b6b1c44514cca1a4718c4a4c65951fa20a3337d01796d"} Oct 30 00:43:01.125133 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:01.125088 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xmsbl" event={"ID":"0b22701a-4358-4fd4-92d9-bd838f5af629","Type":"ContainerStarted","Data":"01c112ff76552443b9e39c5b79208ec96ef2aa7f3e6b7573a69ed107b9f71355"} Oct 30 00:43:01.126706 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:01.126676 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jpp4m" event={"ID":"653fb098-e2a1-4674-9baa-6975e9dd0fec","Type":"ContainerStarted","Data":"874ddcd3a5bb7eb513a1939007b192044d8483cb5e1d5b6603821d7adfc204e6"} Oct 30 00:43:01.747154 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:01.747115 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:43:01.763486 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:01.763461 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kcfzd\"" Oct 30 00:43:01.777369 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:01.777346 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 00:43:02.077303 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.077229 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:02.085303 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.085228 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:02.133740 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.133649 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xmsbl" event={"ID":"0b22701a-4358-4fd4-92d9-bd838f5af629","Type":"ContainerStarted","Data":"824d25d26dbad5bb44010248feedcfc3141bd1dddbb501fd290d873a18c1a526"} Oct 30 00:43:02.160401 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.160364 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:02.230566 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.230529 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7"] Oct 30 00:43:02.233902 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.233875 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7" Oct 30 00:43:02.252020 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.251903 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 00:43:02.252020 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.251946 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-789mk\"" Oct 30 00:43:02.252020 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.251992 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 00:43:02.252337 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.251999 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 00:43:02.252543 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.252522 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 00:43:02.252776 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.252740 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 00:43:02.252776 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.252754 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7"] Oct 30 00:43:02.253596 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.253570 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:02.330566 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.330489 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:02.339439 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.339407 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c78afce-0bbb-49e1-aceb-ae4a69844075-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-g9xp7\" (UID: \"3c78afce-0bbb-49e1-aceb-ae4a69844075\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7" Oct 30 00:43:02.339648 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.339448 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwqqr\" (UniqueName: \"kubernetes.io/projected/3c78afce-0bbb-49e1-aceb-ae4a69844075-kube-api-access-bwqqr\") pod \"prometheus-operator-9cd89b74c-g9xp7\" (UID: \"3c78afce-0bbb-49e1-aceb-ae4a69844075\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7" Oct 30 00:43:02.339648 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.339482 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c78afce-0bbb-49e1-aceb-ae4a69844075-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-g9xp7\" (UID: \"3c78afce-0bbb-49e1-aceb-ae4a69844075\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7" Oct 30 00:43:02.339787 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.339660 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c78afce-0bbb-49e1-aceb-ae4a69844075-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-g9xp7\" (UID: \"3c78afce-0bbb-49e1-aceb-ae4a69844075\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7" Oct 30 00:43:02.441072 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.441017 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c78afce-0bbb-49e1-aceb-ae4a69844075-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-g9xp7\" (UID: \"3c78afce-0bbb-49e1-aceb-ae4a69844075\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7" Oct 30 00:43:02.441269 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.441075 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c78afce-0bbb-49e1-aceb-ae4a69844075-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-g9xp7\" (UID: \"3c78afce-0bbb-49e1-aceb-ae4a69844075\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7" Oct 30 00:43:02.441269 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.441109 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwqqr\" (UniqueName: \"kubernetes.io/projected/3c78afce-0bbb-49e1-aceb-ae4a69844075-kube-api-access-bwqqr\") pod \"prometheus-operator-9cd89b74c-g9xp7\" (UID: \"3c78afce-0bbb-49e1-aceb-ae4a69844075\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7" Oct 30 00:43:02.441269 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.441145 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c78afce-0bbb-49e1-aceb-ae4a69844075-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-g9xp7\" (UID: \"3c78afce-0bbb-49e1-aceb-ae4a69844075\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7" Oct 30 00:43:02.442433 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.442410 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c78afce-0bbb-49e1-aceb-ae4a69844075-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-g9xp7\" (UID: \"3c78afce-0bbb-49e1-aceb-ae4a69844075\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7" Oct 30 00:43:02.446218 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.446169 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c78afce-0bbb-49e1-aceb-ae4a69844075-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-g9xp7\" (UID: \"3c78afce-0bbb-49e1-aceb-ae4a69844075\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7" Oct 30 00:43:02.450481 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.450457 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c78afce-0bbb-49e1-aceb-ae4a69844075-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-g9xp7\" (UID: \"3c78afce-0bbb-49e1-aceb-ae4a69844075\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7" Oct 30 00:43:02.463237 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.463212 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwqqr\" (UniqueName: \"kubernetes.io/projected/3c78afce-0bbb-49e1-aceb-ae4a69844075-kube-api-access-bwqqr\") pod \"prometheus-operator-9cd89b74c-g9xp7\" (UID: \"3c78afce-0bbb-49e1-aceb-ae4a69844075\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7" Oct 30 00:43:02.550999 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.550968 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7" Oct 30 00:43:02.740579 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.740523 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cbd5c8c4-fbqjc"] Oct 30 00:43:02.743793 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.743768 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:02.764595 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.764566 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cbd5c8c4-fbqjc"] Oct 30 00:43:02.774372 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.765346 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vhshc\"" Oct 30 00:43:02.774523 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.765408 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 00:43:02.774634 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.765467 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 00:43:02.774713 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.766119 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 00:43:02.774805 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.766190 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 00:43:02.774924 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.766244 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 00:43:02.774924 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.766300 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 00:43:02.774924 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.774297 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 00:43:02.844841 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.844686 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffa2afe9-832d-452f-855d-982f9e47a601-tmp\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:02.844841 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.844736 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-service-ca\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:02.845034 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.844863 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ffa2afe9-832d-452f-855d-982f9e47a601-console-oauth-config\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:02.845034 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.844891 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-console-config\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:02.845034 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.844911 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ffa2afe9-832d-452f-855d-982f9e47a601-console-serving-cert\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:02.845034 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.844944 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-oauth-serving-cert\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:02.845034 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.844968 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mplm\" (UniqueName: \"kubernetes.io/projected/ffa2afe9-832d-452f-855d-982f9e47a601-kube-api-access-8mplm\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:02.871165 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.871139 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7"] Oct 30 00:43:02.872670 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:43:02.872645 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c78afce_0bbb_49e1_aceb_ae4a69844075.slice/crio-ab4805c524196e761de616228d521027344068919a2c839e780a27bb20f5276c WatchSource:0}: Error finding container ab4805c524196e761de616228d521027344068919a2c839e780a27bb20f5276c: Status 404 returned error can't find the container with id ab4805c524196e761de616228d521027344068919a2c839e780a27bb20f5276c Oct 30 00:43:02.945298 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.945274 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-service-ca\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:02.945416 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.945318 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ffa2afe9-832d-452f-855d-982f9e47a601-console-oauth-config\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:02.945416 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.945346 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-console-config\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:02.945416 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.945368 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ffa2afe9-832d-452f-855d-982f9e47a601-console-serving-cert\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:02.945416 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.945392 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-oauth-serving-cert\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:02.945416 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.945412 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mplm\" (UniqueName: \"kubernetes.io/projected/ffa2afe9-832d-452f-855d-982f9e47a601-kube-api-access-8mplm\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:02.945657 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.945481 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffa2afe9-832d-452f-855d-982f9e47a601-tmp\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:02.945867 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.945839 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffa2afe9-832d-452f-855d-982f9e47a601-tmp\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:02.946023 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.946001 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-service-ca\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:02.946324 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.946296 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-console-config\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:02.946402 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.946339 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-oauth-serving-cert\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:02.948047 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.948026 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ffa2afe9-832d-452f-855d-982f9e47a601-console-oauth-config\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:02.948503 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.948483 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ffa2afe9-832d-452f-855d-982f9e47a601-console-serving-cert\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:02.965578 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:02.965554 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mplm\" (UniqueName: \"kubernetes.io/projected/ffa2afe9-832d-452f-855d-982f9e47a601-kube-api-access-8mplm\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:03.063689 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:03.063653 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:03.139394 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:03.139360 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5jhd7" event={"ID":"81258c01-d653-4a05-87ca-8373e7fad83d","Type":"ContainerStarted","Data":"9b4da3eb24ad866220cc5e79c61da325971895060430ce2ea85f9d5236efce2d"} Oct 30 00:43:03.139585 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:03.139429 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5jhd7" Oct 30 00:43:03.139585 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:03.139457 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5jhd7" event={"ID":"81258c01-d653-4a05-87ca-8373e7fad83d","Type":"ContainerStarted","Data":"72262a6554c5754806b04ed8c5b94699b1244267c13b021ce1c2fd11ab09605a"} Oct 30 00:43:03.140799 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:03.140751 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7" event={"ID":"3c78afce-0bbb-49e1-aceb-ae4a69844075","Type":"ContainerStarted","Data":"ab4805c524196e761de616228d521027344068919a2c839e780a27bb20f5276c"} Oct 30 00:43:03.143158 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:03.143125 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xmsbl" event={"ID":"0b22701a-4358-4fd4-92d9-bd838f5af629","Type":"ContainerStarted","Data":"a00a367c7641f4e93830f78b690894f765d543248f8a50e3f73d3dc8f6b8962b"} Oct 30 00:43:03.185676 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:03.185622 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-5jhd7" podStartSLOduration=2.250043713 podStartE2EDuration="4.185606739s" totalImagesPullingTime="1.935563026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:03.184924469 +0000 UTC m=+55.058655621" watchObservedRunningTime="2026-10-30 00:43:03.185606739 +0000 UTC m=+55.059337890" Oct 30 00:43:03.228516 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:03.228463 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xmsbl" podStartSLOduration=2.057806603 podStartE2EDuration="4.228443963s" totalImagesPullingTime="2.17063736s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:42:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:03.227643544 +0000 UTC m=+55.101374692" watchObservedRunningTime="2026-10-30 00:43:03.228443963 +0000 UTC m=+55.102175115" Oct 30 00:43:03.280626 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:03.280589 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:03.358684 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:03.358656 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cbd5c8c4-fbqjc"] Oct 30 00:43:03.359701 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:03.359676 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:03.362063 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:43:03.362033 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa2afe9_832d_452f_855d_982f9e47a601.slice/crio-9b3abef169ea86e60b9acf2249c3cb82e1023cf6d9e242acaa4994c5af54e583 WatchSource:0}: Error finding container 9b3abef169ea86e60b9acf2249c3cb82e1023cf6d9e242acaa4994c5af54e583: Status 404 returned error can't find the container with id 9b3abef169ea86e60b9acf2249c3cb82e1023cf6d9e242acaa4994c5af54e583 Oct 30 00:43:03.726332 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:03.726299 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:03.828810 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:03.828729 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:04.147794 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:04.147744 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7" event={"ID":"3c78afce-0bbb-49e1-aceb-ae4a69844075","Type":"ContainerStarted","Data":"280b3909feedf5a1523b36a835a080a1513cb9160e81cfc5d08d1572dec0be03"} Oct 30 00:43:04.149831 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:04.149717 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jpp4m" event={"ID":"653fb098-e2a1-4674-9baa-6975e9dd0fec","Type":"ContainerStarted","Data":"af7548b34bfdcc9a511ab7232611e603eb916fadf43bbd8a0fffdc5cad3511ff"} Oct 30 00:43:04.151800 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:04.151755 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cbd5c8c4-fbqjc" event={"ID":"ffa2afe9-832d-452f-855d-982f9e47a601","Type":"ContainerStarted","Data":"9b3abef169ea86e60b9acf2249c3cb82e1023cf6d9e242acaa4994c5af54e583"} Oct 30 00:43:04.208095 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:04.208045 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jpp4m" podStartSLOduration=2.485602212 podStartE2EDuration="5.208026468s" totalImagesPullingTime="2.722424256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:04.207410645 +0000 UTC m=+56.081141799" watchObservedRunningTime="2026-10-30 00:43:04.208026468 +0000 UTC m=+56.081757618" Oct 30 00:43:04.773772 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:04.773739 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:05.161504 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:05.161409 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7" event={"ID":"3c78afce-0bbb-49e1-aceb-ae4a69844075","Type":"ContainerStarted","Data":"b460f6c808de13ce0bd6d887caea42a13912a5eba475e0833954aa5900765b28"} Oct 30 00:43:05.202458 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:05.202407 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7" podStartSLOduration=1.9928759820000002 podStartE2EDuration="3.202391588s" totalImagesPullingTime="1.209515606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:05.202290434 +0000 UTC m=+57.076021591" watchObservedRunningTime="2026-10-30 00:43:05.202391588 +0000 UTC m=+57.076122740" Oct 30 00:43:06.013660 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.013631 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:06.114650 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.114626 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:06.166112 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.166069 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cbd5c8c4-fbqjc" event={"ID":"ffa2afe9-832d-452f-855d-982f9e47a601","Type":"ContainerStarted","Data":"428cc127c34e536430a5519d7f3bb47121127e337a6706d13c5722e96ae9062b"} Oct 30 00:43:06.209820 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.209743 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6cbd5c8c4-fbqjc" podStartSLOduration=1.494488047 podStartE2EDuration="4.209730026s" totalImagesPullingTime="2.715241979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:06.209088664 +0000 UTC m=+58.082819814" watchObservedRunningTime="2026-10-30 00:43:06.209730026 +0000 UTC m=+58.083461177" Oct 30 00:43:06.555401 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.555366 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:06.655690 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.655654 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:06.755405 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.755375 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:06.861419 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.861339 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:06.873398 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.873372 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vngsd"] Oct 30 00:43:06.876595 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.876578 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:06.888112 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.888090 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcf5056b-ef4e-430b-8b5c-83fad65af440-metrics-client-ca\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:06.888221 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.888122 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-accelerators-collector-config\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:06.888221 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.888198 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fcf5056b-ef4e-430b-8b5c-83fad65af440-root\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:06.888285 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.888238 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:06.888285 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.888258 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-tls\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:06.888341 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.888311 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kghwp\" (UniqueName: \"kubernetes.io/projected/fcf5056b-ef4e-430b-8b5c-83fad65af440-kube-api-access-kghwp\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:06.888373 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.888347 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-textfile\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:06.888401 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.888376 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcf5056b-ef4e-430b-8b5c-83fad65af440-sys\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:06.888499 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.888484 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-wtmp\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:06.946025 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.945986 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:06.946328 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.946314 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 00:43:06.946599 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.946583 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 00:43:06.946683 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.946660 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rbt7f\"" Oct 30 00:43:06.946817 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.946802 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 00:43:06.988829 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.988796 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-wtmp\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:06.988829 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.988831 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcf5056b-ef4e-430b-8b5c-83fad65af440-metrics-client-ca\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:06.989034 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.988850 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-accelerators-collector-config\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:06.989034 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.988880 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fcf5056b-ef4e-430b-8b5c-83fad65af440-root\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:06.989034 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.988897 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:06.989034 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.988963 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fcf5056b-ef4e-430b-8b5c-83fad65af440-root\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:06.989034 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.989002 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-wtmp\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:06.989034 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.989017 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-tls\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:06.989321 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.989057 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kghwp\" (UniqueName: \"kubernetes.io/projected/fcf5056b-ef4e-430b-8b5c-83fad65af440-kube-api-access-kghwp\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:06.989321 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.989090 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-textfile\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:06.989321 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.989140 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcf5056b-ef4e-430b-8b5c-83fad65af440-sys\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:06.989321 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.989239 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcf5056b-ef4e-430b-8b5c-83fad65af440-sys\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:06.989521 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.989413 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-textfile\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:06.989521 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.989460 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-accelerators-collector-config\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:06.989627 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.989538 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcf5056b-ef4e-430b-8b5c-83fad65af440-metrics-client-ca\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:06.991281 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.991262 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-tls\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:06.991373 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:06.991309 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:07.009157 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:07.009122 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kghwp\" (UniqueName: \"kubernetes.io/projected/fcf5056b-ef4e-430b-8b5c-83fad65af440-kube-api-access-kghwp\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:07.148858 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:07.148780 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:07.192557 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:07.192528 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:07.203461 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:43:07.203435 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcf5056b_ef4e_430b_8b5c_83fad65af440.slice/crio-3494cae29415aae77bc7ed41918f4dc56688122704debb112eb50231cc64e9ba WatchSource:0}: Error finding container 3494cae29415aae77bc7ed41918f4dc56688122704debb112eb50231cc64e9ba: Status 404 returned error can't find the container with id 3494cae29415aae77bc7ed41918f4dc56688122704debb112eb50231cc64e9ba Oct 30 00:43:07.458214 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:07.458127 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:08.026645 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:08.026613 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:08.127234 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:08.127200 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:08.172867 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:08.172831 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vngsd" event={"ID":"fcf5056b-ef4e-430b-8b5c-83fad65af440","Type":"ContainerStarted","Data":"3494cae29415aae77bc7ed41918f4dc56688122704debb112eb50231cc64e9ba"} Oct 30 00:43:08.293531 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:08.293465 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:09.177570 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:09.177541 2573 generic.go:356] "Generic (PLEG): container finished" podID="fcf5056b-ef4e-430b-8b5c-83fad65af440" containerID="276deba1049dbbb6552b4db239bc59fa20bc594e454a7a3464eddd05f25602dd" exitCode=0 Oct 30 00:43:09.177719 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:09.177584 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vngsd" event={"ID":"fcf5056b-ef4e-430b-8b5c-83fad65af440","Type":"ContainerDied","Data":"276deba1049dbbb6552b4db239bc59fa20bc594e454a7a3464eddd05f25602dd"} Oct 30 00:43:10.083716 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:10.083686 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:10.181963 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:10.181937 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:10.182875 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:10.182844 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vngsd" event={"ID":"fcf5056b-ef4e-430b-8b5c-83fad65af440","Type":"ContainerStarted","Data":"8eb68c4944fd40ff0540d7fc9549a9ce93a479211e0f19f34a18e35ca8f62164"} Oct 30 00:43:10.182994 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:10.182927 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vngsd" event={"ID":"fcf5056b-ef4e-430b-8b5c-83fad65af440","Type":"ContainerStarted","Data":"fc937e6a3fea9333c2baa572270512a14ff2b874ce6a8bedd938ab55e5a24ed6"} Oct 30 00:43:10.225472 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:10.225423 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vngsd" podStartSLOduration=3.154820918 podStartE2EDuration="4.225408906s" totalImagesPullingTime="1.001569293s" totalInitContainerRuntime="69.018695ms" isStatefulPod=true podCreationTimestamp="2026-10-30 00:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:10.224725962 +0000 UTC m=+62.098457112" watchObservedRunningTime="2026-10-30 00:43:10.225408906 +0000 UTC m=+62.099140056" Oct 30 00:43:10.280732 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:10.280695 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:11.103479 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:11.103444 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:11.191621 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:11.191590 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:11.647698 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:11.647650 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:11.716477 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:11.716447 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-vmstj"] Oct 30 00:43:11.719688 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:11.719667 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmstj" Oct 30 00:43:11.724601 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:11.724575 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/67bde315-b2ea-4d17-88d8-f290cd2c994e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vmstj\" (UID: \"67bde315-b2ea-4d17-88d8-f290cd2c994e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmstj" Oct 30 00:43:11.724720 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:11.724619 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp68j\" (UniqueName: \"kubernetes.io/projected/67bde315-b2ea-4d17-88d8-f290cd2c994e-kube-api-access-mp68j\") pod \"monitoring-plugin-77f4f76594-vmstj\" (UID: \"67bde315-b2ea-4d17-88d8-f290cd2c994e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmstj" Oct 30 00:43:11.740118 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:11.740091 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-vmstj"] Oct 30 00:43:11.741635 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:11.741612 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:11.752690 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:11.752670 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 00:43:11.752950 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:11.752936 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-hz7cw\"" Oct 30 00:43:11.825712 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:11.825684 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/67bde315-b2ea-4d17-88d8-f290cd2c994e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vmstj\" (UID: \"67bde315-b2ea-4d17-88d8-f290cd2c994e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmstj" Oct 30 00:43:11.825846 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:11.825716 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp68j\" (UniqueName: \"kubernetes.io/projected/67bde315-b2ea-4d17-88d8-f290cd2c994e-kube-api-access-mp68j\") pod \"monitoring-plugin-77f4f76594-vmstj\" (UID: \"67bde315-b2ea-4d17-88d8-f290cd2c994e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmstj" Oct 30 00:43:11.828148 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:11.828132 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/67bde315-b2ea-4d17-88d8-f290cd2c994e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vmstj\" (UID: \"67bde315-b2ea-4d17-88d8-f290cd2c994e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmstj" Oct 30 00:43:11.846075 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:11.846046 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp68j\" (UniqueName: \"kubernetes.io/projected/67bde315-b2ea-4d17-88d8-f290cd2c994e-kube-api-access-mp68j\") pod \"monitoring-plugin-77f4f76594-vmstj\" (UID: \"67bde315-b2ea-4d17-88d8-f290cd2c994e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmstj" Oct 30 00:43:11.892715 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:11.892678 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:12.030223 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:12.030190 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmstj" Oct 30 00:43:12.326054 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:12.325930 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-vmstj"] Oct 30 00:43:12.328667 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:43:12.328642 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67bde315_b2ea_4d17_88d8_f290cd2c994e.slice/crio-004ba09d25f8f7a4344e34113018b282226d51e97b12c38e6fc6cf1d24b021f8 WatchSource:0}: Error finding container 004ba09d25f8f7a4344e34113018b282226d51e97b12c38e6fc6cf1d24b021f8: Status 404 returned error can't find the container with id 004ba09d25f8f7a4344e34113018b282226d51e97b12c38e6fc6cf1d24b021f8 Oct 30 00:43:13.064087 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.064054 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:13.064385 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.064113 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:13.069491 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.069465 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:13.154097 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.154069 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5jhd7" Oct 30 00:43:13.194177 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.194130 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmstj" event={"ID":"67bde315-b2ea-4d17-88d8-f290cd2c994e","Type":"ContainerStarted","Data":"004ba09d25f8f7a4344e34113018b282226d51e97b12c38e6fc6cf1d24b021f8"} Oct 30 00:43:13.198251 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.198226 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:13.337355 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.337273 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-metrics-certs\") pod \"network-metrics-daemon-przs7\" (UID: \"1c2f1e93-f19e-4c0f-944e-7f52261f6e81\") " pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:43:13.337355 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.337340 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-original-pull-secret\") pod \"global-pull-secret-syncer-xdkdr\" (UID: \"b0661ca3-6f0a-4744-9d50-f2baa1a80b45\") " pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:43:13.352808 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.352745 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 00:43:13.352981 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.352818 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 00:43:13.361467 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.361443 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-metrics-certs\") pod \"network-metrics-daemon-przs7\" (UID: \"1c2f1e93-f19e-4c0f-944e-7f52261f6e81\") " pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:43:13.361581 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.361509 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-original-pull-secret\") pod \"global-pull-secret-syncer-xdkdr\" (UID: \"b0661ca3-6f0a-4744-9d50-f2baa1a80b45\") " pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:43:13.382775 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.382731 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vdsv4\"" Oct 30 00:43:13.386844 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.386814 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:43:13.439848 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.439806 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4gkr\" (UniqueName: \"kubernetes.io/projected/b3f55f18-4143-4806-b9e4-989e153f7e38-kube-api-access-m4gkr\") pod \"network-check-target-c4b2t\" (UID: \"b3f55f18-4143-4806-b9e4-989e153f7e38\") " pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:43:13.455210 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.455185 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 00:43:13.475216 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.475197 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 00:43:13.477751 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.477721 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kcfzd\"" Oct 30 00:43:13.485043 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.485022 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:43:13.487860 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.487838 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4gkr\" (UniqueName: \"kubernetes.io/projected/b3f55f18-4143-4806-b9e4-989e153f7e38-kube-api-access-m4gkr\") pod \"network-check-target-c4b2t\" (UID: \"b3f55f18-4143-4806-b9e4-989e153f7e38\") " pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:43:13.677523 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.677497 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-p4p9c\"" Oct 30 00:43:13.683040 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.683021 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:43:13.693177 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.693153 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xdkdr"] Oct 30 00:43:13.767285 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.767234 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:13.775992 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.775965 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 00:43:13.781666 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.781644 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.800192 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.800168 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-przs7"] Oct 30 00:43:13.801514 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.801494 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 00:43:13.801822 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.801799 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 00:43:13.801933 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.801835 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 00:43:13.801933 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.801840 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dzgqx\"" Oct 30 00:43:13.801933 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.801877 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 00:43:13.801933 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.801930 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 00:43:13.802631 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:43:13.802605 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2f1e93_f19e_4c0f_944e_7f52261f6e81.slice/crio-6037462f1e78bb1f94b06d5412aa46efec347619ee0909ac5800f8efa81d3f45 WatchSource:0}: Error finding container 6037462f1e78bb1f94b06d5412aa46efec347619ee0909ac5800f8efa81d3f45: Status 404 returned error can't find the container with id 6037462f1e78bb1f94b06d5412aa46efec347619ee0909ac5800f8efa81d3f45 Oct 30 00:43:13.812818 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.812701 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 00:43:13.812818 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.812780 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 00:43:13.812818 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.812783 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 00:43:13.815238 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.815221 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 00:43:13.817190 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.817170 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 00:43:13.821046 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.820752 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 00:43:13.821046 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.820890 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 00:43:13.821046 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.820935 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 00:43:13.843094 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.843063 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 00:43:13.843234 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.843116 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 00:43:13.849586 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.849558 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.849690 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.849587 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.849690 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.849606 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.849690 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.849628 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-web-config\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.849815 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.849716 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.849849 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.849822 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e991db07-d748-4ced-8d7a-4266194e89d8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.849880 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.849859 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.849915 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.849904 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.849946 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.849925 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.849986 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.849971 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.850036 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.850025 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.850068 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.850049 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mghhs\" (UniqueName: \"kubernetes.io/projected/e991db07-d748-4ced-8d7a-4266194e89d8-kube-api-access-mghhs\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.850104 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.850071 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.850104 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.850089 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.850162 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.850131 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.850193 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.850184 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-config\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.850231 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.850216 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.850267 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.850239 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.850298 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.850265 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e991db07-d748-4ced-8d7a-4266194e89d8-config-out\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.850338 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.850326 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.901841 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.901788 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 00:43:13.902015 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.901944 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8uop28j2hbr7f\"" Oct 30 00:43:13.950691 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.950612 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.950691 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.950644 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.950691 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.950663 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.950991 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.950872 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.950991 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.950915 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-web-config\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.950991 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.950953 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.950991 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.950979 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e991db07-d748-4ced-8d7a-4266194e89d8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.951204 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.951004 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.951204 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.951037 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.951204 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.951075 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.951204 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.951111 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.951204 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.951147 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.951204 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.951168 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mghhs\" (UniqueName: \"kubernetes.io/projected/e991db07-d748-4ced-8d7a-4266194e89d8-kube-api-access-mghhs\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.951204 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.951200 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.951559 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.951227 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.951559 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.951252 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.951559 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.951283 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-config\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.951559 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.951309 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.951559 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.951338 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.951559 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.951372 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e991db07-d748-4ced-8d7a-4266194e89d8-config-out\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.951559 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.951403 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.951559 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.951472 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.952162 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.952135 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.952900 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.952876 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.953298 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.953280 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.954253 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.954225 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.954438 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.954414 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e991db07-d748-4ced-8d7a-4266194e89d8-config-out\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.954552 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.954496 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-web-config\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.954872 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.954853 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.955053 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.955015 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e991db07-d748-4ced-8d7a-4266194e89d8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.955053 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.955039 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.956267 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.956243 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.957089 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.957056 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.957184 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.957126 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.957249 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.957237 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.957453 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.957435 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.958179 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.958161 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.958271 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.958254 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-config\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.958321 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.958270 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.971070 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.971050 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mghhs\" (UniqueName: \"kubernetes.io/projected/e991db07-d748-4ced-8d7a-4266194e89d8-kube-api-access-mghhs\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:13.979966 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:13.979946 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c4b2t"] Oct 30 00:43:13.982699 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:43:13.982677 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f55f18_4143_4806_b9e4_989e153f7e38.slice/crio-557a69cba7345b9287acd09ba0cbdc66d6c50244e6730aa9908eab1505bd1617 WatchSource:0}: Error finding container 557a69cba7345b9287acd09ba0cbdc66d6c50244e6730aa9908eab1505bd1617: Status 404 returned error can't find the container with id 557a69cba7345b9287acd09ba0cbdc66d6c50244e6730aa9908eab1505bd1617 Oct 30 00:43:14.114198 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.114158 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:14.203281 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.203174 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmstj" event={"ID":"67bde315-b2ea-4d17-88d8-f290cd2c994e","Type":"ContainerStarted","Data":"d40ba06f917650f7a04fd16bbc1e5f748d0dc2b425cb815fbdd78629e312f3e9"} Oct 30 00:43:14.203424 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.203390 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmstj" Oct 30 00:43:14.204607 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.204578 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xdkdr" event={"ID":"b0661ca3-6f0a-4744-9d50-f2baa1a80b45","Type":"ContainerStarted","Data":"fad95519ff0692168a93889a183f5b95dbe2f7c1378dd82b2fa81af27995108d"} Oct 30 00:43:14.209746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.209703 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmstj" Oct 30 00:43:14.210785 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.210705 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c4b2t" event={"ID":"b3f55f18-4143-4806-b9e4-989e153f7e38","Type":"ContainerStarted","Data":"557a69cba7345b9287acd09ba0cbdc66d6c50244e6730aa9908eab1505bd1617"} Oct 30 00:43:14.212527 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.212498 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-przs7" event={"ID":"1c2f1e93-f19e-4c0f-944e-7f52261f6e81","Type":"ContainerStarted","Data":"6037462f1e78bb1f94b06d5412aa46efec347619ee0909ac5800f8efa81d3f45"} Oct 30 00:43:14.246116 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.246064 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmstj" podStartSLOduration=1.9022639190000001 podStartE2EDuration="3.246049475s" totalImagesPullingTime="1.343785556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:14.245456527 +0000 UTC m=+66.119187677" watchObservedRunningTime="2026-10-30 00:43:14.246049475 +0000 UTC m=+66.119780625" Oct 30 00:43:14.380058 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.378781 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-8698cc7897-jflvm"] Oct 30 00:43:14.382185 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.382162 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:14.401040 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.401003 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 00:43:14.402985 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.402923 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8698cc7897-jflvm"] Oct 30 00:43:14.438636 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.438599 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 00:43:14.445962 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:43:14.445920 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode991db07_d748_4ced_8d7a_4266194e89d8.slice/crio-cc6e866c85861f00cee7ad73ec83efa452092d9d609585d1e4be909d96416e2c WatchSource:0}: Error finding container cc6e866c85861f00cee7ad73ec83efa452092d9d609585d1e4be909d96416e2c: Status 404 returned error can't find the container with id cc6e866c85861f00cee7ad73ec83efa452092d9d609585d1e4be909d96416e2c Oct 30 00:43:14.456485 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.456426 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-tmp\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:14.456614 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.456487 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2726g\" (UniqueName: \"kubernetes.io/projected/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-kube-api-access-2726g\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:14.456614 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.456537 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-config\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:14.456614 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.456564 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-serving-cert\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:14.456614 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.456608 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-service-ca\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:14.456746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.456628 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-trusted-ca-bundle\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:14.456746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.456678 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-oauth-config\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:14.456746 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.456700 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-oauth-serving-cert\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:14.557439 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.557393 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-config\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:14.557615 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.557447 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-serving-cert\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:14.557615 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.557492 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-service-ca\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:14.557615 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.557515 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-trusted-ca-bundle\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:14.557615 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.557561 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-oauth-config\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:14.557615 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.557582 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-oauth-serving-cert\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:14.557615 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.557608 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-tmp\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:14.557931 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.557647 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2726g\" (UniqueName: \"kubernetes.io/projected/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-kube-api-access-2726g\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:14.558698 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.558670 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-config\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:14.559576 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.559527 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-tmp\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:14.559723 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.559699 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-oauth-serving-cert\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:14.560044 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.560027 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-service-ca\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:14.560112 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.560065 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-trusted-ca-bundle\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:14.561534 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.561506 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-serving-cert\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:14.562959 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.562912 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-oauth-config\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:14.581833 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.581803 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2726g\" (UniqueName: \"kubernetes.io/projected/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-kube-api-access-2726g\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:14.710771 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:14.710677 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:15.025801 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:15.025225 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8698cc7897-jflvm"] Oct 30 00:43:15.031833 ip-10-0-140-206 kubenswrapper[2573]: W1030 00:43:15.031802 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e33ac0_f034_477b_8e5a_aaf82a23d7fa.slice/crio-a39b78a775f0f6b5a51dd6831da2352abac4740bed387f3af7c58b4c2ef8676c WatchSource:0}: Error finding container a39b78a775f0f6b5a51dd6831da2352abac4740bed387f3af7c58b4c2ef8676c: Status 404 returned error can't find the container with id a39b78a775f0f6b5a51dd6831da2352abac4740bed387f3af7c58b4c2ef8676c Oct 30 00:43:15.218466 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:15.218430 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-przs7" event={"ID":"1c2f1e93-f19e-4c0f-944e-7f52261f6e81","Type":"ContainerStarted","Data":"3e79bcc627ab6c619ffd3cc30c7bed0d2cf3cef65a30589a160833bf1393f88d"} Oct 30 00:43:15.223857 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:15.223122 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8698cc7897-jflvm" event={"ID":"07e33ac0-f034-477b-8e5a-aaf82a23d7fa","Type":"ContainerStarted","Data":"5f7166b9f2cffea47388c8a6a93e4cb2979ecf3e0663fa7fc6d1740de0ef55d6"} Oct 30 00:43:15.223857 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:15.223801 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8698cc7897-jflvm" event={"ID":"07e33ac0-f034-477b-8e5a-aaf82a23d7fa","Type":"ContainerStarted","Data":"a39b78a775f0f6b5a51dd6831da2352abac4740bed387f3af7c58b4c2ef8676c"} Oct 30 00:43:15.226943 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:15.226910 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e991db07-d748-4ced-8d7a-4266194e89d8","Type":"ContainerStarted","Data":"cc6e866c85861f00cee7ad73ec83efa452092d9d609585d1e4be909d96416e2c"} Oct 30 00:43:15.267792 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:15.267730 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-8698cc7897-jflvm" podStartSLOduration=1.2677165879999999 podStartE2EDuration="1.267716588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:15.265887979 +0000 UTC m=+67.139619132" watchObservedRunningTime="2026-10-30 00:43:15.267716588 +0000 UTC m=+67.141447718" Oct 30 00:43:16.213082 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:16.213045 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:16.233399 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:16.232965 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-przs7" event={"ID":"1c2f1e93-f19e-4c0f-944e-7f52261f6e81","Type":"ContainerStarted","Data":"3e54aaa4e63613ba2dd8719213d4e592023b42c533d46ccf78377431dbcfae1c"} Oct 30 00:43:16.234995 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:16.234969 2573 generic.go:356] "Generic (PLEG): container finished" podID="e991db07-d748-4ced-8d7a-4266194e89d8" containerID="4cde129fff51071d51c81973829d7b60cbdc27630f79782f7aa3fcb590cfe88f" exitCode=0 Oct 30 00:43:16.235118 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:16.235048 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e991db07-d748-4ced-8d7a-4266194e89d8","Type":"ContainerDied","Data":"4cde129fff51071d51c81973829d7b60cbdc27630f79782f7aa3fcb590cfe88f"} Oct 30 00:43:16.275198 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:16.275148 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-przs7" podStartSLOduration=66.092771682 podStartE2EDuration="1m7.275134918s" totalImagesPullingTime="1.182363236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:16.274730244 +0000 UTC m=+68.148461395" watchObservedRunningTime="2026-10-30 00:43:16.275134918 +0000 UTC m=+68.148866069" Oct 30 00:43:16.316822 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:16.316785 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:18.247576 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:18.247540 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c4b2t" event={"ID":"b3f55f18-4143-4806-b9e4-989e153f7e38","Type":"ContainerStarted","Data":"8cdf03b00feb68fbe2754e0fa76ca136f2037ae421e6d9565460a9c32e294616"} Oct 30 00:43:18.248073 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:18.247901 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:43:18.731844 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:18.731808 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:18.833661 ip-10-0-140-206 kubenswrapper[2573]: I1030 00:43:18.833549 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:19.617318 ip-10-0-140-206 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 00:43:19.625312 ip-10-0-140-206 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 00:43:19.625484 ip-10-0-140-206 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 00:43:19.625533 ip-10-0-140-206 systemd[1]: kubelet.service: Consumed 3.937s CPU time. Oct 30 00:43:19.635418 ip-10-0-140-206 systemd[1]: Starting Kubernetes Kubelet... Oct 30 00:43:19.719456 ip-10-0-140-206 kubenswrapper[5300]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:43:19.719456 ip-10-0-140-206 kubenswrapper[5300]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 00:43:19.719456 ip-10-0-140-206 kubenswrapper[5300]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:43:19.719456 ip-10-0-140-206 kubenswrapper[5300]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 00:43:19.721985 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.721893 5300 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:43:19.721985 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.721980 5300 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:43:19.721985 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.721985 5300 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:43:19.721985 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.721989 5300 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:43:19.722252 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.721995 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:43:19.722252 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722000 5300 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:43:19.722252 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722004 5300 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:43:19.722252 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722008 5300 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:43:19.722252 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722012 5300 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:43:19.722252 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722016 5300 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:43:19.722252 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722020 5300 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:43:19.722252 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722024 5300 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:43:19.722252 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722028 5300 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:43:19.722252 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722032 5300 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:43:19.722252 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722036 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:43:19.722252 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722040 5300 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:43:19.722252 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722044 5300 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:43:19.722252 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722048 5300 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:43:19.722252 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722052 5300 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:43:19.722252 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722056 5300 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:43:19.722252 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722060 5300 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:43:19.722252 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722064 5300 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:43:19.722252 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722070 5300 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:43:19.723100 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722077 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:43:19.723100 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722082 5300 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:43:19.723100 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722086 5300 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:43:19.723100 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722092 5300 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:43:19.723100 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722101 5300 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:43:19.723100 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722106 5300 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:43:19.723100 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722110 5300 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:43:19.723100 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722115 5300 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:43:19.723100 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722120 5300 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:43:19.723100 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722124 5300 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:43:19.723100 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722128 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:43:19.723100 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722133 5300 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:43:19.723100 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722137 5300 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:43:19.723100 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722142 5300 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:43:19.723100 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722147 5300 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:43:19.723100 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722151 5300 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:43:19.723100 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722155 5300 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:43:19.723100 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722159 5300 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:43:19.723100 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722163 5300 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:43:19.723100 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722168 5300 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:43:19.724036 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722172 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:43:19.724036 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722176 5300 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:43:19.724036 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722181 5300 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:43:19.724036 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722186 5300 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:43:19.724036 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722190 5300 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:43:19.724036 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722194 5300 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:43:19.724036 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722198 5300 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:43:19.724036 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722202 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:43:19.724036 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722207 5300 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:43:19.724036 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722211 5300 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:43:19.724036 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722217 5300 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:43:19.724036 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722222 5300 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:43:19.724036 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722226 5300 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:43:19.724036 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722230 5300 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:43:19.724036 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722235 5300 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:43:19.724036 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722239 5300 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:43:19.724036 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722244 5300 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:43:19.724036 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722248 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:43:19.724036 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722255 5300 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:43:19.724036 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722261 5300 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:43:19.724932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722265 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:43:19.724932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722269 5300 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:43:19.724932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722274 5300 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:43:19.724932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722278 5300 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:43:19.724932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722283 5300 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:43:19.724932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722289 5300 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:43:19.724932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722293 5300 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:43:19.724932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722298 5300 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:43:19.724932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722302 5300 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:43:19.724932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722306 5300 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:43:19.724932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722311 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:43:19.724932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722315 5300 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:43:19.724932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722319 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:43:19.724932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722323 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:43:19.724932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722328 5300 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:43:19.724932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722333 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:43:19.724932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722338 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:43:19.724932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722343 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:43:19.724932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722347 5300 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:43:19.725782 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722351 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:43:19.725782 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722355 5300 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:43:19.725782 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722359 5300 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:43:19.725782 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722364 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:43:19.725782 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722368 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:43:19.725782 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722373 5300 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:43:19.725782 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722377 5300 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:43:19.725782 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722381 5300 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:43:19.725782 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722386 5300 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:43:19.725782 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722391 5300 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:43:19.725782 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722395 5300 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:43:19.725782 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722402 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:43:19.725782 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722407 5300 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:43:19.725782 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722412 5300 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:43:19.725782 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722417 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:43:19.725782 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722421 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:43:19.725782 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722425 5300 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:43:19.725782 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722432 5300 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:43:19.725782 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722436 5300 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:43:19.726641 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722440 5300 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:43:19.726641 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722445 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:43:19.726641 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722449 5300 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:43:19.726641 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722454 5300 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:43:19.726641 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.722458 5300 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:43:19.726641 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723732 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:43:19.726641 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723743 5300 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:43:19.726641 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723748 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:43:19.726641 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723753 5300 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:43:19.726641 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723776 5300 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:43:19.726641 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723785 5300 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:43:19.726641 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723789 5300 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:43:19.726641 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723794 5300 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:43:19.726641 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723798 5300 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:43:19.726641 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723803 5300 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:43:19.726641 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723807 5300 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:43:19.726641 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723812 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:43:19.726641 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723816 5300 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:43:19.726641 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723821 5300 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:43:19.726641 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723826 5300 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:43:19.727542 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723830 5300 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:43:19.727542 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723834 5300 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:43:19.727542 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723839 5300 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:43:19.727542 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723844 5300 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:43:19.727542 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723848 5300 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:43:19.727542 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723853 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:43:19.727542 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723857 5300 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:43:19.727542 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723862 5300 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:43:19.727542 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723866 5300 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:43:19.727542 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723871 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:43:19.727542 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723875 5300 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:43:19.727542 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723882 5300 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:43:19.727542 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723889 5300 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:43:19.727542 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723894 5300 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:43:19.727542 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723898 5300 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:43:19.727542 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723903 5300 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:43:19.727542 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723907 5300 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:43:19.727542 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723912 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:43:19.727542 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723916 5300 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:43:19.728374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723920 5300 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:43:19.728374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723924 5300 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:43:19.728374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723930 5300 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:43:19.728374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723934 5300 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:43:19.728374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723939 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:43:19.728374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723943 5300 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:43:19.728374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723948 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:43:19.728374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723953 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:43:19.728374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723958 5300 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:43:19.728374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723964 5300 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:43:19.728374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723968 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:43:19.728374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723973 5300 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:43:19.728374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723977 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:43:19.728374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723981 5300 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:43:19.728374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723986 5300 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:43:19.728374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723991 5300 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:43:19.728374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.723995 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:43:19.728374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724000 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:43:19.728374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724004 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:43:19.728374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724009 5300 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:43:19.729004 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724013 5300 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:43:19.729004 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724018 5300 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:43:19.729004 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724023 5300 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:43:19.729004 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724027 5300 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:43:19.729004 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724031 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:43:19.729004 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724035 5300 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:43:19.729004 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724039 5300 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:43:19.729004 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724043 5300 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:43:19.729004 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724047 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:43:19.729004 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724051 5300 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:43:19.729004 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724055 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:43:19.729004 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724060 5300 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:43:19.729004 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724064 5300 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:43:19.729004 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724068 5300 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:43:19.729004 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724073 5300 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:43:19.729004 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724077 5300 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:43:19.729004 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724082 5300 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:43:19.729004 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724087 5300 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:43:19.729004 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724091 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:43:19.729004 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724096 5300 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:43:19.729499 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724101 5300 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:43:19.729499 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724105 5300 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:43:19.729499 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724109 5300 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:43:19.729499 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724113 5300 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:43:19.729499 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724118 5300 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:43:19.729499 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724122 5300 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:43:19.729499 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724127 5300 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:43:19.729499 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724132 5300 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:43:19.729499 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724136 5300 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:43:19.729499 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724141 5300 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:43:19.729499 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724145 5300 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:43:19.729499 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724149 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:43:19.729499 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724154 5300 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:43:19.729499 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724159 5300 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:43:19.729499 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724163 5300 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:43:19.729499 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724168 5300 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:43:19.729499 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724172 5300 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:43:19.729499 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724176 5300 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:43:19.729499 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724181 5300 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:43:19.729499 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724185 5300 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:43:19.730022 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724189 5300 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:43:19.730022 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724194 5300 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:43:19.730022 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724198 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:43:19.730022 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724203 5300 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:43:19.730022 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724207 5300 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:43:19.730022 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724212 5300 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:43:19.730022 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724216 5300 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:43:19.730022 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724221 5300 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:43:19.730022 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724225 5300 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:43:19.730022 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724229 5300 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:43:19.730022 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724236 5300 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:43:19.730022 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.724241 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:43:19.730022 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725118 5300 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:43:19.730022 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725128 5300 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:43:19.730022 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725133 5300 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:43:19.730022 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725137 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:43:19.730022 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725144 5300 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:43:19.730022 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725148 5300 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:43:19.730532 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725153 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:43:19.730532 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725158 5300 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:43:19.730532 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725162 5300 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:43:19.730532 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725167 5300 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:43:19.730532 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725172 5300 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:43:19.730532 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725177 5300 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:43:19.730532 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725181 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:43:19.730532 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725185 5300 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:43:19.730532 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725190 5300 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:43:19.730532 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725194 5300 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:43:19.730532 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725199 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:43:19.730532 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725203 5300 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:43:19.730532 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725207 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:43:19.730532 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725211 5300 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:43:19.730532 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725215 5300 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:43:19.730532 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725220 5300 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:43:19.730532 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725224 5300 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:43:19.730532 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725228 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:43:19.730532 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725233 5300 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:43:19.730532 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725240 5300 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:43:19.731059 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725244 5300 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:43:19.731059 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725249 5300 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:43:19.731059 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725253 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:43:19.731059 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725258 5300 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:43:19.731059 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725264 5300 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:43:19.731059 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725268 5300 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:43:19.731059 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725273 5300 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:43:19.731059 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725278 5300 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:43:19.731059 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725282 5300 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:43:19.731059 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725286 5300 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:43:19.731059 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725290 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:43:19.731059 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725297 5300 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:43:19.731059 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725303 5300 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:43:19.731059 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725308 5300 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:43:19.731059 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725313 5300 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:43:19.731059 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725318 5300 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:43:19.731059 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725323 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:43:19.731059 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725328 5300 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:43:19.731059 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725332 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:43:19.731059 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725337 5300 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:43:19.731527 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725341 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:43:19.731527 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725345 5300 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:43:19.731527 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725349 5300 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:43:19.731527 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725353 5300 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:43:19.731527 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725358 5300 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:43:19.731527 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725362 5300 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:43:19.731527 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725367 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:43:19.731527 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725371 5300 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:43:19.731527 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725377 5300 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:43:19.731527 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725382 5300 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:43:19.731527 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725386 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:43:19.731527 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725392 5300 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:43:19.731527 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725396 5300 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:43:19.731527 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725402 5300 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:43:19.731527 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725407 5300 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:43:19.731527 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725411 5300 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:43:19.731527 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725415 5300 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:43:19.731527 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725421 5300 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:43:19.731527 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725425 5300 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:43:19.731527 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725429 5300 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:43:19.732056 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725433 5300 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:43:19.732056 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725437 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:43:19.732056 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725442 5300 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:43:19.732056 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725446 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:43:19.732056 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725451 5300 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:43:19.732056 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725455 5300 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:43:19.732056 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725459 5300 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:43:19.732056 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725463 5300 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:43:19.732056 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725468 5300 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:43:19.732056 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725472 5300 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:43:19.732056 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725476 5300 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:43:19.732056 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725480 5300 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:43:19.732056 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725484 5300 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:43:19.732056 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725489 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:43:19.732056 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725493 5300 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:43:19.732056 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725498 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:43:19.732056 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725502 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:43:19.732056 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725506 5300 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:43:19.732056 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725511 5300 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:43:19.732525 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725515 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:43:19.732525 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725519 5300 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:43:19.732525 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725524 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:43:19.732525 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725528 5300 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:43:19.732525 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725535 5300 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:43:19.732525 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725540 5300 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:43:19.732525 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725546 5300 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:43:19.732525 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725551 5300 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:43:19.732525 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725555 5300 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:43:19.732525 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725561 5300 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:43:19.732525 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725568 5300 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:43:19.732525 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725577 5300 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:43:19.732525 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725582 5300 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:43:19.732525 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725587 5300 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:43:19.732525 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725593 5300 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:43:19.732525 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725598 5300 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:43:19.732525 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725603 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:43:19.732525 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725607 5300 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:43:19.732525 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725612 5300 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:43:19.733005 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725616 5300 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:43:19.733005 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.725620 5300 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:43:19.733005 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725753 5300 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 00:43:19.733005 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725780 5300 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 00:43:19.733005 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725807 5300 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 00:43:19.733005 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725814 5300 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 00:43:19.733005 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725822 5300 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 00:43:19.733005 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725828 5300 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 00:43:19.733005 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725834 5300 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 00:43:19.733005 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725840 5300 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 00:43:19.733005 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725846 5300 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 00:43:19.733005 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725851 5300 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 00:43:19.733005 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725856 5300 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 00:43:19.733005 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725862 5300 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 00:43:19.733005 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725867 5300 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 00:43:19.733005 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725871 5300 flags.go:64] FLAG: --cgroup-root="" Oct 30 00:43:19.733005 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725876 5300 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 00:43:19.733005 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725885 5300 flags.go:64] FLAG: --client-ca-file="" Oct 30 00:43:19.733005 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725890 5300 flags.go:64] FLAG: --cloud-provider="external" Oct 30 00:43:19.733005 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725896 5300 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 00:43:19.733005 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725904 5300 flags.go:64] FLAG: --cluster-domain="" Oct 30 00:43:19.733005 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725908 5300 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 00:43:19.733005 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725913 5300 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 00:43:19.733582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725919 5300 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 00:43:19.733582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725924 5300 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 00:43:19.733582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725931 5300 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 00:43:19.733582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725936 5300 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 00:43:19.733582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725942 5300 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 00:43:19.733582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725947 5300 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 00:43:19.733582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725952 5300 flags.go:64] FLAG: --contention-profiling="false" Oct 30 00:43:19.733582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725956 5300 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 00:43:19.733582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725962 5300 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 00:43:19.733582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725967 5300 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 00:43:19.733582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725972 5300 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 00:43:19.733582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725979 5300 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 00:43:19.733582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725984 5300 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 00:43:19.733582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725989 5300 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 00:43:19.733582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725994 5300 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 00:43:19.733582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.725999 5300 flags.go:64] FLAG: --enable-server="true" Oct 30 00:43:19.733582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726004 5300 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 00:43:19.733582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726010 5300 flags.go:64] FLAG: --event-burst="100" Oct 30 00:43:19.733582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726015 5300 flags.go:64] FLAG: --event-qps="50" Oct 30 00:43:19.733582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726020 5300 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 00:43:19.733582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726025 5300 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 00:43:19.733582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726030 5300 flags.go:64] FLAG: --eviction-hard="" Oct 30 00:43:19.733582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726036 5300 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 00:43:19.733582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726041 5300 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 00:43:19.734248 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726046 5300 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 00:43:19.734248 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726051 5300 flags.go:64] FLAG: --eviction-soft="" Oct 30 00:43:19.734248 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726058 5300 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 00:43:19.734248 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726063 5300 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 00:43:19.734248 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726069 5300 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 00:43:19.734248 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726074 5300 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 00:43:19.734248 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726079 5300 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 00:43:19.734248 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726083 5300 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 00:43:19.734248 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726088 5300 flags.go:64] FLAG: --feature-gates="" Oct 30 00:43:19.734248 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726096 5300 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 00:43:19.734248 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726101 5300 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 00:43:19.734248 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726107 5300 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 00:43:19.734248 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726112 5300 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 00:43:19.734248 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726117 5300 flags.go:64] FLAG: --healthz-port="10248" Oct 30 00:43:19.734248 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726122 5300 flags.go:64] FLAG: --help="false" Oct 30 00:43:19.734248 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726127 5300 flags.go:64] FLAG: --hostname-override="ip-10-0-140-206.ec2.internal" Oct 30 00:43:19.734248 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726132 5300 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 00:43:19.734248 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726137 5300 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 00:43:19.734248 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726142 5300 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 00:43:19.734248 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726148 5300 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 00:43:19.734248 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726154 5300 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 00:43:19.734248 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726159 5300 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 00:43:19.734248 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726163 5300 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 00:43:19.734837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726168 5300 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 00:43:19.734837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726173 5300 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 00:43:19.734837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726177 5300 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 00:43:19.734837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726183 5300 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 00:43:19.734837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726188 5300 flags.go:64] FLAG: --kube-reserved="" Oct 30 00:43:19.734837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726193 5300 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 00:43:19.734837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726197 5300 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 00:43:19.734837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726203 5300 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 00:43:19.734837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726207 5300 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 00:43:19.734837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726212 5300 flags.go:64] FLAG: --lock-file="" Oct 30 00:43:19.734837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726216 5300 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 00:43:19.734837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726223 5300 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 00:43:19.734837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726228 5300 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 00:43:19.734837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726239 5300 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 00:43:19.734837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726244 5300 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 00:43:19.734837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726249 5300 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 00:43:19.734837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726253 5300 flags.go:64] FLAG: --logging-format="text" Oct 30 00:43:19.734837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726258 5300 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 00:43:19.734837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726264 5300 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 00:43:19.734837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726268 5300 flags.go:64] FLAG: --manifest-url="" Oct 30 00:43:19.734837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726274 5300 flags.go:64] FLAG: --manifest-url-header="" Oct 30 00:43:19.734837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726282 5300 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 00:43:19.734837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726287 5300 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 00:43:19.734837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726293 5300 flags.go:64] FLAG: --max-pods="110" Oct 30 00:43:19.734837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726299 5300 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 00:43:19.735445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726304 5300 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 00:43:19.735445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726308 5300 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 00:43:19.735445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726313 5300 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 00:43:19.735445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726319 5300 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 00:43:19.735445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726324 5300 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 00:43:19.735445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726329 5300 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 00:43:19.735445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726339 5300 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 00:43:19.735445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726344 5300 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 00:43:19.735445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726349 5300 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 00:43:19.735445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726354 5300 flags.go:64] FLAG: --pod-cidr="" Oct 30 00:43:19.735445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726359 5300 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 00:43:19.735445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726364 5300 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 00:43:19.735445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726368 5300 flags.go:64] FLAG: --pods-per-core="0" Oct 30 00:43:19.735445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726373 5300 flags.go:64] FLAG: --port="10250" Oct 30 00:43:19.735445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726378 5300 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 00:43:19.735445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726383 5300 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e015b2d1ed84cca5" Oct 30 00:43:19.735445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726388 5300 flags.go:64] FLAG: --qos-reserved="" Oct 30 00:43:19.735445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726393 5300 flags.go:64] FLAG: --read-only-port="10255" Oct 30 00:43:19.735445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726400 5300 flags.go:64] FLAG: --register-node="true" Oct 30 00:43:19.735445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726406 5300 flags.go:64] FLAG: --register-with-taints="" Oct 30 00:43:19.735445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726414 5300 flags.go:64] FLAG: --registry-burst="10" Oct 30 00:43:19.735445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726419 5300 flags.go:64] FLAG: --registry-qps="5" Oct 30 00:43:19.735445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726423 5300 flags.go:64] FLAG: --reserved-cpus="" Oct 30 00:43:19.735445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726428 5300 flags.go:64] FLAG: --reserved-memory="" Oct 30 00:43:19.735445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726434 5300 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 00:43:19.735445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726440 5300 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 00:43:19.736089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726445 5300 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 00:43:19.736089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726450 5300 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 00:43:19.736089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726455 5300 flags.go:64] FLAG: --runonce="false" Oct 30 00:43:19.736089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726460 5300 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 00:43:19.736089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726465 5300 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 00:43:19.736089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726471 5300 flags.go:64] FLAG: --seccomp-default="false" Oct 30 00:43:19.736089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726475 5300 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 00:43:19.736089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726480 5300 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 00:43:19.736089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726486 5300 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 00:43:19.736089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726491 5300 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 00:43:19.736089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726496 5300 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 00:43:19.736089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726500 5300 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 00:43:19.736089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726505 5300 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 00:43:19.736089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726510 5300 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 00:43:19.736089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726515 5300 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 00:43:19.736089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726520 5300 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 00:43:19.736089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726526 5300 flags.go:64] FLAG: --system-cgroups="" Oct 30 00:43:19.736089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726531 5300 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 00:43:19.736089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726539 5300 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 00:43:19.736089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726544 5300 flags.go:64] FLAG: --tls-cert-file="" Oct 30 00:43:19.736089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726549 5300 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 00:43:19.736089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726555 5300 flags.go:64] FLAG: --tls-min-version="" Oct 30 00:43:19.736089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726560 5300 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 00:43:19.736089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726565 5300 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 00:43:19.736089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726572 5300 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 00:43:19.736676 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726577 5300 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 00:43:19.736676 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726583 5300 flags.go:64] FLAG: --v="2" Oct 30 00:43:19.736676 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726591 5300 flags.go:64] FLAG: --version="false" Oct 30 00:43:19.736676 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726597 5300 flags.go:64] FLAG: --vmodule="" Oct 30 00:43:19.736676 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726603 5300 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 00:43:19.736676 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.726609 5300 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 00:43:19.736676 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.726893 5300 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:43:19.736676 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.726901 5300 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:43:19.736676 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.726907 5300 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:43:19.736676 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.726912 5300 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:43:19.736676 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.726916 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:43:19.736676 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.726928 5300 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:43:19.736676 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.726933 5300 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:43:19.736676 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.726937 5300 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:43:19.736676 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.726942 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:43:19.736676 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.726946 5300 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:43:19.736676 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.726950 5300 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:43:19.736676 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.726955 5300 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:43:19.736676 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.726959 5300 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:43:19.736676 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.726964 5300 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:43:19.736676 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.726968 5300 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:43:19.737345 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.726972 5300 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:43:19.737345 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.726976 5300 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:43:19.737345 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.726981 5300 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:43:19.737345 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.726985 5300 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:43:19.737345 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.726990 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:43:19.737345 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.726994 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:43:19.737345 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.726999 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:43:19.737345 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727004 5300 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:43:19.737345 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727008 5300 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:43:19.737345 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727012 5300 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:43:19.737345 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727018 5300 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:43:19.737345 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727022 5300 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:43:19.737345 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727028 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:43:19.737345 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727033 5300 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:43:19.737345 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727037 5300 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:43:19.737345 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727042 5300 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:43:19.737345 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727046 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:43:19.737345 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727050 5300 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:43:19.737345 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727054 5300 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:43:19.737345 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727059 5300 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:43:19.738255 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727063 5300 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:43:19.738255 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727068 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:43:19.738255 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727072 5300 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:43:19.738255 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727087 5300 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:43:19.738255 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727092 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:43:19.738255 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727096 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:43:19.738255 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727101 5300 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:43:19.738255 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727105 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:43:19.738255 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727109 5300 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:43:19.738255 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727113 5300 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:43:19.738255 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727118 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:43:19.738255 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727122 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:43:19.738255 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727126 5300 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:43:19.738255 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727131 5300 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:43:19.738255 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727135 5300 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:43:19.738255 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727139 5300 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:43:19.738255 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727144 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:43:19.738255 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727148 5300 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:43:19.738255 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727152 5300 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:43:19.739065 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727156 5300 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:43:19.739065 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727161 5300 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:43:19.739065 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727165 5300 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:43:19.739065 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727171 5300 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:43:19.739065 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727176 5300 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:43:19.739065 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727181 5300 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:43:19.739065 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727186 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:43:19.739065 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727190 5300 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:43:19.739065 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727194 5300 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:43:19.739065 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727198 5300 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:43:19.739065 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727203 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:43:19.739065 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727207 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:43:19.739065 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727212 5300 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:43:19.739065 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727216 5300 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:43:19.739065 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727221 5300 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:43:19.739065 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727225 5300 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:43:19.739065 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727229 5300 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:43:19.739065 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727242 5300 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:43:19.739065 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727247 5300 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:43:19.739599 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727252 5300 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:43:19.739599 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727256 5300 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:43:19.739599 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727261 5300 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:43:19.739599 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727266 5300 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:43:19.739599 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727270 5300 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:43:19.739599 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727282 5300 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:43:19.739599 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727287 5300 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:43:19.739599 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727292 5300 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:43:19.739599 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727296 5300 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:43:19.739599 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727303 5300 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:43:19.739599 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727310 5300 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:43:19.739599 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727316 5300 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:43:19.739599 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727321 5300 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:43:19.739599 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727326 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:43:19.739599 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727331 5300 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:43:19.739599 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727336 5300 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:43:19.739599 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727342 5300 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:43:19.739599 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727347 5300 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:43:19.739599 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727353 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:43:19.740153 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727359 5300 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:43:19.740153 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727364 5300 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:43:19.740153 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727369 5300 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:43:19.740153 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727373 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:43:19.740153 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727378 5300 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:43:19.740153 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727382 5300 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:43:19.740153 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727386 5300 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:43:19.740153 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727391 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:43:19.740153 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727395 5300 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:43:19.740153 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727399 5300 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:43:19.740153 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727404 5300 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:43:19.740153 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727408 5300 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:43:19.740153 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727421 5300 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:43:19.740153 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727426 5300 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:43:19.740153 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.727467 5300 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:43:19.740153 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727543 5300 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:43:19.740628 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727551 5300 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:43:19.740628 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727555 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:43:19.740628 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727560 5300 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:43:19.740628 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727564 5300 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:43:19.740628 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727569 5300 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:43:19.740628 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727573 5300 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:43:19.740628 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727578 5300 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:43:19.740628 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727582 5300 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:43:19.740628 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727586 5300 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:43:19.740628 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727591 5300 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:43:19.740628 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727595 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:43:19.740628 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727600 5300 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:43:19.740628 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727604 5300 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:43:19.740628 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727609 5300 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:43:19.740628 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727616 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:43:19.740628 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727621 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:43:19.740628 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727625 5300 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:43:19.740628 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727630 5300 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:43:19.740628 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727634 5300 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:43:19.741189 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727638 5300 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:43:19.741189 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727642 5300 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:43:19.741189 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727648 5300 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:43:19.741189 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727652 5300 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:43:19.741189 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727657 5300 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:43:19.741189 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727661 5300 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:43:19.741189 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727666 5300 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:43:19.741189 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727671 5300 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:43:19.741189 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727675 5300 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:43:19.741189 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727679 5300 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:43:19.741189 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727692 5300 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:43:19.741189 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727696 5300 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:43:19.741189 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727701 5300 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:43:19.741189 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727707 5300 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:43:19.741189 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727725 5300 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:43:19.741189 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727729 5300 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:43:19.741189 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727734 5300 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:43:19.741189 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727739 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:43:19.741189 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727743 5300 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:43:19.741189 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727748 5300 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:43:19.741708 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727752 5300 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:43:19.741708 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727773 5300 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:43:19.741708 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727778 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:43:19.741708 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727784 5300 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:43:19.741708 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727791 5300 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:43:19.741708 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727795 5300 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:43:19.741708 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727800 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:43:19.741708 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727806 5300 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:43:19.741708 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727811 5300 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:43:19.741708 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727815 5300 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:43:19.741708 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727820 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:43:19.741708 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727824 5300 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:43:19.741708 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727828 5300 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:43:19.741708 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727833 5300 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:43:19.741708 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727837 5300 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:43:19.741708 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727842 5300 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:43:19.741708 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727846 5300 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:43:19.741708 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727850 5300 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:43:19.741708 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727855 5300 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:43:19.742234 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727859 5300 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:43:19.742234 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727863 5300 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:43:19.742234 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727868 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:43:19.742234 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727877 5300 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:43:19.742234 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727889 5300 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:43:19.742234 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727893 5300 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:43:19.742234 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727897 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:43:19.742234 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727901 5300 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:43:19.742234 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727906 5300 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:43:19.742234 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727910 5300 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:43:19.742234 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727915 5300 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:43:19.742234 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727919 5300 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:43:19.742234 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727924 5300 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:43:19.742234 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727928 5300 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:43:19.742234 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727933 5300 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:43:19.742234 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727937 5300 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:43:19.742234 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727941 5300 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:43:19.742234 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727945 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:43:19.742234 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727950 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:43:19.742234 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727954 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:43:19.742945 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727963 5300 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:43:19.742945 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727967 5300 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:43:19.742945 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727972 5300 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:43:19.742945 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727976 5300 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:43:19.742945 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727981 5300 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:43:19.742945 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727985 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:43:19.742945 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727989 5300 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:43:19.742945 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727994 5300 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:43:19.742945 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.727998 5300 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:43:19.742945 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.728002 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:43:19.742945 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.728007 5300 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:43:19.742945 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.728011 5300 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:43:19.742945 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.728016 5300 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:43:19.742945 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.728020 5300 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:43:19.742945 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.728026 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:43:19.742945 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.728030 5300 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:43:19.742945 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.728034 5300 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:43:19.742945 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.728047 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:43:19.742945 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.728051 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:43:19.742945 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.728056 5300 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:43:19.743482 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.728060 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:43:19.743482 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.728064 5300 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:43:19.743482 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.728068 5300 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:43:19.743482 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.728073 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:43:19.743482 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.728077 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:43:19.743482 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.728082 5300 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:43:19.743482 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.728086 5300 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:43:19.743482 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.732534 5300 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 00:43:19.743482 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.732549 5300 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 00:43:19.743482 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.732567 5300 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 00:43:19.743482 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.732572 5300 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 00:43:19.743482 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732645 5300 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:43:19.743482 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732650 5300 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:43:19.743482 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732654 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:43:19.743482 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732658 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:43:19.743482 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732661 5300 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:43:19.743482 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732664 5300 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:43:19.743482 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732667 5300 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:43:19.743482 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732670 5300 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:43:19.743482 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732673 5300 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:43:19.745051 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732675 5300 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:43:19.745051 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732678 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:43:19.745051 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732680 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:43:19.745051 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732683 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:43:19.745051 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732687 5300 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:43:19.745051 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732691 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:43:19.745051 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732695 5300 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:43:19.745051 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732698 5300 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:43:19.745051 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732702 5300 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:43:19.745051 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732712 5300 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:43:19.745051 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732715 5300 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:43:19.745051 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732718 5300 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:43:19.745051 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732721 5300 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:43:19.745051 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732723 5300 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:43:19.745051 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732726 5300 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:43:19.745051 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732729 5300 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:43:19.745051 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732732 5300 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:43:19.745051 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732734 5300 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:43:19.745051 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732737 5300 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:43:19.745517 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732740 5300 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:43:19.745517 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732743 5300 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:43:19.745517 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732746 5300 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:43:19.745517 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732748 5300 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:43:19.745517 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732751 5300 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:43:19.745517 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732754 5300 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:43:19.745517 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732756 5300 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:43:19.745517 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732774 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:43:19.745517 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732778 5300 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:43:19.745517 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732782 5300 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:43:19.745517 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732786 5300 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:43:19.745517 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732788 5300 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:43:19.745517 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732791 5300 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:43:19.745517 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732793 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:43:19.745517 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732796 5300 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:43:19.745517 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732799 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:43:19.745517 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732802 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:43:19.745517 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732804 5300 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:43:19.745517 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732814 5300 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:43:19.746006 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732817 5300 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:43:19.746006 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732819 5300 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:43:19.746006 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732822 5300 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:43:19.746006 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732824 5300 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:43:19.746006 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732833 5300 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:43:19.746006 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732836 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:43:19.746006 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732839 5300 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:43:19.746006 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732841 5300 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:43:19.746006 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732844 5300 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:43:19.746006 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732847 5300 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:43:19.746006 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732850 5300 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:43:19.746006 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732853 5300 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:43:19.746006 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732856 5300 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:43:19.746006 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732858 5300 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:43:19.746006 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732861 5300 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:43:19.746006 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732864 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:43:19.746006 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732866 5300 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:43:19.746006 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732869 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:43:19.746006 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732872 5300 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:43:19.746006 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732874 5300 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:43:19.746520 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732877 5300 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:43:19.746520 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732879 5300 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:43:19.746520 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732882 5300 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:43:19.746520 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732885 5300 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:43:19.746520 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732887 5300 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:43:19.746520 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732892 5300 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:43:19.746520 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732896 5300 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:43:19.746520 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732899 5300 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:43:19.746520 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732902 5300 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:43:19.746520 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732906 5300 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:43:19.746520 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732909 5300 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:43:19.746520 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732912 5300 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:43:19.746520 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732914 5300 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:43:19.746520 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732917 5300 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:43:19.746520 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732920 5300 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:43:19.746520 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732922 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:43:19.746520 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732925 5300 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:43:19.746520 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732933 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:43:19.746520 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732936 5300 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:43:19.746520 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732939 5300 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:43:19.747018 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732942 5300 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:43:19.747018 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732945 5300 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:43:19.747018 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732947 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:43:19.747018 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732950 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:43:19.747018 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732953 5300 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:43:19.747018 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732956 5300 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:43:19.747018 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732959 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:43:19.747018 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732961 5300 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:43:19.747018 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732964 5300 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:43:19.747018 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732966 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:43:19.747018 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732969 5300 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:43:19.747018 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732972 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:43:19.747018 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732974 5300 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:43:19.747018 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732977 5300 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:43:19.747018 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732979 5300 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:43:19.747018 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732982 5300 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:43:19.747018 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732985 5300 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:43:19.747018 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732987 5300 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:43:19.747018 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.732990 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:43:19.747481 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.733017 5300 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:43:19.747481 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733206 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:43:19.747481 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733211 5300 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:43:19.747481 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733214 5300 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:43:19.747481 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733216 5300 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:43:19.747481 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733219 5300 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:43:19.747481 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733222 5300 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:43:19.747481 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733226 5300 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:43:19.747481 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733230 5300 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:43:19.747481 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733233 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:43:19.747481 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733236 5300 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:43:19.747481 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733244 5300 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:43:19.747481 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733247 5300 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:43:19.747481 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733250 5300 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:43:19.747481 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733252 5300 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:43:19.747481 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733255 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:43:19.747883 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733258 5300 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:43:19.747883 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733260 5300 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:43:19.747883 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733263 5300 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:43:19.747883 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733266 5300 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:43:19.747883 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733268 5300 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:43:19.747883 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733271 5300 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:43:19.747883 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733274 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:43:19.747883 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733276 5300 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:43:19.747883 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733280 5300 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:43:19.747883 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733284 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:43:19.747883 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733287 5300 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:43:19.747883 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733290 5300 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:43:19.747883 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733293 5300 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:43:19.747883 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733296 5300 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:43:19.747883 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733299 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:43:19.747883 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733302 5300 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:43:19.747883 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733304 5300 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:43:19.747883 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733307 5300 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:43:19.747883 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733309 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:43:19.748374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733312 5300 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:43:19.748374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733314 5300 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:43:19.748374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733317 5300 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:43:19.748374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733320 5300 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:43:19.748374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733323 5300 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:43:19.748374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733325 5300 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:43:19.748374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733328 5300 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:43:19.748374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733331 5300 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:43:19.748374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733333 5300 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:43:19.748374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733341 5300 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:43:19.748374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733344 5300 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:43:19.748374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733347 5300 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:43:19.748374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733350 5300 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:43:19.748374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733353 5300 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:43:19.748374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733355 5300 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:43:19.748374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733358 5300 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:43:19.748374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733361 5300 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:43:19.748374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733363 5300 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:43:19.748374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733366 5300 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:43:19.748374 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733369 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:43:19.748929 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733371 5300 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:43:19.748929 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733374 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:43:19.748929 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733376 5300 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:43:19.748929 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733379 5300 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:43:19.748929 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733381 5300 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:43:19.748929 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733384 5300 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:43:19.748929 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733386 5300 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:43:19.748929 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733389 5300 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:43:19.748929 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733392 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:43:19.748929 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733395 5300 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:43:19.748929 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733397 5300 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:43:19.748929 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733400 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:43:19.748929 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733402 5300 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:43:19.748929 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733405 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:43:19.748929 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733408 5300 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:43:19.748929 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733410 5300 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:43:19.748929 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733413 5300 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:43:19.748929 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733416 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:43:19.748929 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733418 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:43:19.749390 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733421 5300 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:43:19.749390 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733424 5300 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:43:19.749390 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733426 5300 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:43:19.749390 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733434 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:43:19.749390 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733437 5300 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:43:19.749390 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733440 5300 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:43:19.749390 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733442 5300 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:43:19.749390 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733445 5300 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:43:19.749390 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733448 5300 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:43:19.749390 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733450 5300 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:43:19.749390 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733453 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:43:19.749390 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733455 5300 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:43:19.749390 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733458 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:43:19.749390 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733461 5300 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:43:19.749390 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733463 5300 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:43:19.749390 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733466 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:43:19.749390 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733468 5300 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:43:19.749390 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733471 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:43:19.749390 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733473 5300 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:43:19.750272 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733476 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:43:19.750272 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733479 5300 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:43:19.750272 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733481 5300 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:43:19.750272 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733484 5300 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:43:19.750272 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733486 5300 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:43:19.750272 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733489 5300 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:43:19.750272 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733492 5300 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:43:19.750272 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733494 5300 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:43:19.750272 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733497 5300 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:43:19.750272 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733499 5300 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:43:19.750272 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733502 5300 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:43:19.750272 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733505 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:43:19.750272 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733507 5300 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:43:19.750272 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733510 5300 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:43:19.750272 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.733532 5300 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 00:43:19.750272 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733578 5300 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 00:43:19.750981 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733583 5300 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 00:43:19.750981 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733593 5300 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 00:43:19.750981 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733597 5300 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 00:43:19.750981 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733600 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 00:43:19.750981 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733603 5300 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 00:43:19.750981 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733606 5300 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 00:43:19.750981 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733612 5300 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 00:43:19.750981 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733615 5300 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 00:43:19.750981 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733617 5300 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 00:43:19.750981 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733620 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 00:43:19.750981 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733623 5300 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 00:43:19.750981 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733625 5300 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 00:43:19.750981 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733628 5300 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 00:43:19.750981 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733630 5300 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 00:43:19.750981 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733633 5300 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 00:43:19.750981 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733636 5300 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 00:43:19.750981 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733638 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 00:43:19.750981 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733641 5300 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 00:43:19.750981 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733645 5300 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 00:43:19.751775 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733648 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 00:43:19.751775 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733651 5300 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 00:43:19.751775 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733654 5300 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 00:43:19.751775 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733656 5300 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 00:43:19.751775 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733659 5300 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 00:43:19.751775 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733662 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 00:43:19.751775 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733664 5300 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 00:43:19.751775 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733667 5300 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 00:43:19.751775 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733670 5300 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 00:43:19.751775 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733673 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 00:43:19.751775 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733675 5300 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 00:43:19.751775 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733678 5300 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 00:43:19.751775 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733681 5300 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 00:43:19.751775 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733684 5300 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 00:43:19.751775 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733686 5300 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 00:43:19.751775 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733696 5300 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 00:43:19.751775 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733699 5300 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 00:43:19.751775 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733702 5300 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 00:43:19.751775 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733704 5300 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 00:43:19.751775 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733708 5300 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 00:43:19.752609 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733711 5300 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 00:43:19.752609 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733714 5300 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 00:43:19.752609 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733716 5300 feature_gate.go:386] unrecognized feature gate: Example Oct 30 00:43:19.752609 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733719 5300 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 00:43:19.752609 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733723 5300 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 00:43:19.752609 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733726 5300 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 00:43:19.752609 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733729 5300 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 00:43:19.752609 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733731 5300 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 00:43:19.752609 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733734 5300 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 00:43:19.752609 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733737 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 00:43:19.752609 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733739 5300 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 00:43:19.752609 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733742 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 00:43:19.752609 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733745 5300 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 00:43:19.752609 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733747 5300 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 00:43:19.752609 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733750 5300 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 00:43:19.752609 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733753 5300 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 00:43:19.752609 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733756 5300 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 00:43:19.752609 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733782 5300 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 00:43:19.752609 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733785 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 00:43:19.752609 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733788 5300 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 00:43:19.753240 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733791 5300 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 00:43:19.753240 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733794 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 00:43:19.753240 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733804 5300 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 00:43:19.753240 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733808 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 00:43:19.753240 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733810 5300 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 00:43:19.753240 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733813 5300 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 00:43:19.753240 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733815 5300 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 00:43:19.753240 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733818 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 00:43:19.753240 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733827 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 00:43:19.753240 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733830 5300 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 00:43:19.753240 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733833 5300 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 00:43:19.753240 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733837 5300 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 00:43:19.753240 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733840 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 00:43:19.753240 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733842 5300 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 00:43:19.753240 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733845 5300 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 00:43:19.753240 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733848 5300 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 00:43:19.753240 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733853 5300 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 00:43:19.753240 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733856 5300 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 00:43:19.753240 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733859 5300 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 00:43:19.753932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733861 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 00:43:19.753932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733864 5300 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 00:43:19.753932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733866 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 00:43:19.753932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733869 5300 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 00:43:19.753932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733872 5300 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 00:43:19.753932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733874 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 00:43:19.753932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733877 5300 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 00:43:19.753932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733880 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 00:43:19.753932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733882 5300 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 00:43:19.753932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733885 5300 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 00:43:19.753932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733888 5300 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 00:43:19.753932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733890 5300 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 00:43:19.753932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733893 5300 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 00:43:19.753932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733896 5300 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 00:43:19.753932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733898 5300 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 00:43:19.753932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733901 5300 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 00:43:19.753932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733904 5300 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 00:43:19.753932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733906 5300 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 00:43:19.753932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733909 5300 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 00:43:19.753932 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733911 5300 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 00:43:19.754458 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733914 5300 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 00:43:19.754458 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733916 5300 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 00:43:19.754458 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733925 5300 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 00:43:19.754458 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733928 5300 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 00:43:19.754458 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733932 5300 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 00:43:19.754458 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733935 5300 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 00:43:19.754458 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:43:19.733938 5300 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 00:43:19.754458 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.734002 5300 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 00:43:19.754458 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.735093 5300 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 00:43:19.754458 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.735147 5300 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 00:43:19.754458 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.735342 5300 server.go:1014] "Starting client certificate rotation" Oct 30 00:43:19.754458 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.735487 5300 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 00:43:19.754458 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.735577 5300 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 00:37:09 +0000 UTC" deadline="2028-04-09 15:20:50.706030947 +0000 UTC" Oct 30 00:43:19.754458 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.735635 5300 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12662h37m30.970398168s" Oct 30 00:43:19.754458 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.735916 5300 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:43:19.754458 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.739110 5300 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 00:43:19.754870 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.741087 5300 log.go:25] "Validated CRI v1 runtime API" Oct 30 00:43:19.754870 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.743104 5300 log.go:25] "Validated CRI v1 image API" Oct 30 00:43:19.754870 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.743894 5300 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 00:43:19.754870 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.745870 5300 fs.go:135] Filesystem UUIDs: map[1b29b2ea-801b-49c1-b11e-b5b8de5e0a85:/dev/nvme0n1p4 70925af6-95b4-43ec-834e-1817dce62819:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 00:43:19.754992 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.745893 5300 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/004ba09d25f8f7a4344e34113018b282226d51e97b12c38e6fc6cf1d24b021f8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/004ba09d25f8f7a4344e34113018b282226d51e97b12c38e6fc6cf1d24b021f8/userdata/shm major:0 minor:300 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01c112ff76552443b9e39c5b79208ec96ef2aa7f3e6b7573a69ed107b9f71355/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01c112ff76552443b9e39c5b79208ec96ef2aa7f3e6b7573a69ed107b9f71355/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1a21ea02e80d4f6d786dbd8b2d11fb89a274e7c3ff0c3b317588934510b97e06/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1a21ea02e80d4f6d786dbd8b2d11fb89a274e7c3ff0c3b317588934510b97e06/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3494cae29415aae77bc7ed41918f4dc56688122704debb112eb50231cc64e9ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3494cae29415aae77bc7ed41918f4dc56688122704debb112eb50231cc64e9ba/userdata/shm major:0 minor:284 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4cd9efa42b7843ae35ae9d8eadc51084370007512f89c38570b4c4ecc65a59f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4cd9efa42b7843ae35ae9d8eadc51084370007512f89c38570b4c4ecc65a59f2/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/557a69cba7345b9287acd09ba0cbdc66d6c50244e6730aa9908eab1505bd1617/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/557a69cba7345b9287acd09ba0cbdc66d6c50244e6730aa9908eab1505bd1617/userdata/shm major:0 minor:313 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6037462f1e78bb1f94b06d5412aa46efec347619ee0909ac5800f8efa81d3f45/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6037462f1e78bb1f94b06d5412aa46efec347619ee0909ac5800f8efa81d3f45/userdata/shm major:0 minor:309 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/641e3ef2e21a72d92a5de7c37c8369f43fc34067da733bababa0b8bba76242c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/641e3ef2e21a72d92a5de7c37c8369f43fc34067da733bababa0b8bba76242c1/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/839ff87247ef7ca34d8a7fecffcd2d142d915609c291cc4c257adcda2241eb00/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/839ff87247ef7ca34d8a7fecffcd2d142d915609c291cc4c257adcda2241eb00/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/874ddcd3a5bb7eb513a1939007b192044d8483cb5e1d5b6603821d7adfc204e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/874ddcd3a5bb7eb513a1939007b192044d8483cb5e1d5b6603821d7adfc204e6/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8aedcc0431068aad1e5868f70d8904e299ca29410447c75871583ef000efb149/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8aedcc0431068aad1e5868f70d8904e299ca29410447c75871583ef000efb149/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b3abef169ea86e60b9acf2249c3cb82e1023cf6d9e242acaa4994c5af54e583/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b3abef169ea86e60b9acf2249c3cb82e1023cf6d9e242acaa4994c5af54e583/userdata/shm major:0 minor:250 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ef8f37acadd6669c5af3620533b330d8f1b32b11361b56756eadaf7258e388f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ef8f37acadd6669c5af3620533b330d8f1b32b11361b56756eadaf7258e388f/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a39b78a775f0f6b5a51dd6831da2352abac4740bed387f3af7c58b4c2ef8676c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a39b78a775f0f6b5a51dd6831da2352abac4740bed387f3af7c58b4c2ef8676c/userdata/shm major:0 minor:344 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab4805c524196e761de616228d521027344068919a2c839e780a27bb20f5276c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab4805c524196e761de616228d521027344068919a2c839e780a27bb20f5276c/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b76808fc47c412c7ba94d1e61361c17514fc485ba206f74e99d4d3584f101901/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b76808fc47c412c7ba94d1e61361c17514fc485ba206f74e99d4d3584f101901/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c963d0df141bd6ca560142b1b23b521cdf62b24164a3baab88f5318e8503bffd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c963d0df141bd6ca560142b1b23b521cdf62b24164a3baab88f5318e8503bffd/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc6e866c85861f00cee7ad73ec83efa452092d9d609585d1e4be909d96416e2c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc6e866c85861f00cee7ad73ec83efa452092d9d609585d1e4be909d96416e2c/userdata/shm major:0 minor:337 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cd71fcdd9fcb74d62ca6fd43f9446fd30908000a64453e19d6f2e8de7bafcadd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cd71fcdd9fcb74d62ca6fd43f9446fd30908000a64453e19d6f2e8de7bafcadd/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d9783f5758a10e5b89b5103e04b1261b0cfd92b52c5cf73c558c6b763e53a481/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d9783f5758a10e5b89b5103e04b1261b0cfd92b52c5cf73c558c6b763e53a481/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e23232e117081451795faf1fc0b232b8ae80acdaa22d5d0a82c2d92146c78453/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e23232e117081451795faf1fc0b232b8ae80acdaa22d5d0a82c2d92146c78453/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fad95519ff0692168a93889a183f5b95dbe2f7c1378dd82b2fa81af27995108d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fad95519ff0692168a93889a183f5b95dbe2f7c1378dd82b2fa81af27995108d/userdata/shm major:0 minor:306 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07e33ac0-f034-477b-8e5a-aaf82a23d7fa/volumes/kubernetes.io~projected/kube-api-access-2726g:{mountpoint:/var/lib/kubelet/pods/07e33ac0-f034-477b-8e5a-aaf82a23d7fa/volumes/kubernetes.io~projected/kube-api-access-2726g major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07e33ac0-f034-477b-8e5a-aaf82a23d7fa/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/07e33ac0-f034-477b-8e5a-aaf82a23d7fa/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07e33ac0-f034-477b-8e5a-aaf82a23d7fa/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/07e33ac0-f034-477b-8e5a-aaf82a23d7fa/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b22701a-4358-4fd4-92d9-bd838f5af629/volumes/kubernetes.io~projected/kube-api-access-b9srn:{mountpoint:/var/lib/kubelet/pods/0b22701a-4358-4fd4-92d9-bd838f5af629/volumes/kubernetes.io~projected/kube-api-access-b9srn major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b22701a-4358-4fd4-92d9-bd838f5af629/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/0b22701a-4358-4fd4-92d9-bd838f5af629/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c2f1e93-f19e-4c0f-944e-7f52261f6e81/volumes/kubernetes.io~projected/kube-api-access-9x2hb:{mountpoint:/var/lib/kubelet/pods/1c2f1e93-f19e-4c0f-944e-7f52261f6e81/volumes/kubernetes.io~projected/kube-api-access-9x2hb major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c2f1e93-f19e-4c0f-944e-7f52261f6e81/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/1c2f1e93-f19e-4c0f-944e-7f52261f6e81/volumes/kubernetes.io~secret/metrics-certs major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f1c6441-4ebb-48d6-9e09-bf69e431eb57/volumes/kubernetes.io~projected/kube-api-access-4l7h2:{mountpoint:/var/lib/kubelet/pods/1f1c6441-4ebb-48d6-9e09-bf69e431eb57/volumes/kubernetes.io~projected/kube-api-access-4l7h2 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/271e26db-6a8b-4e46-afb7-139cbc57c1e1/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/271e26db-6a8b-4e46-afb7-139cbc57c1e1/volumes/kubernetes.io~secret/agent-certs major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c78afce-0bbb-49e1-aceb-ae4a69844075/volumes/kubernetes.io~projected/kube-api-access-bwqqr:{mountpoint:/var/lib/kubelet/pods/3c78afce-0bbb-49e1-aceb-ae4a69844075/volumes/kubernetes.io~projected/kube-api-access-bwqqr major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c78afce-0bbb-49e1-aceb-ae4a69844075/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/3c78afce-0bbb-49e1-aceb-ae4a69844075/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c78afce-0bbb-49e1-aceb-ae4a69844075/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/3c78afce-0bbb-49e1-aceb-ae4a69844075/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49f04f3b-49d5-4c22-88a2-e33e106853ce/volumes/kubernetes.io~projected/kube-api-access-tlnvv:{mountpoint:/var/lib/kubelet/pods/49f04f3b-49d5-4c22-88a2-e33e106853ce/volumes/kubernetes.io~projected/kube-api-access-tlnvv major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5dff88d3-2e35-40e9-8aea-208519485adc/volumes/kubernetes.io~projected/kube-api-access-w48mv:{mountpoint:/var/lib/kubelet/pods/5dff88d3-2e35-40e9-8aea-208519485adc/volumes/kubernetes.io~projected/kube-api-access-w48mv major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/653fb098-e2a1-4674-9baa-6975e9dd0fec/volumes/kubernetes.io~projected/kube-api-access-s8kj9:{mountpoint:/var/lib/kubelet/pods/653fb098-e2a1-4674-9baa-6975e9dd0fec/volumes/kubernetes.io~projected/kube-api-access-s8kj9 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/653fb098-e2a1-4674-9baa-6975e9dd0fec/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/653fb098-e2a1-4674-9baa-6975e9dd0fec/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67bde315-b2ea-4d17-88d8-f290cd2c994e/volumes/kubernetes.io~projected/kube-api-access-mp68j:{mountpoint:/var/lib/kubelet/pods/67bde315-b2ea-4d17-88d8-f290cd2c994e/volumes/kubernetes.io~projected/kube-api-access-mp68j major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67bde315-b2ea-4d17-88d8-f290cd2c994e/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/67bde315-b2ea-4d17-88d8-f290cd2c994e/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7127ca6b-3962-47a9-9089-07bcba277bc7/volumes/kubernetes.io~projected/kube-api-access-hj6jk:{mountpoint:/var/lib/kubelet/pods/7127ca6b-3962-47a9-9089-07bcba277bc7/volumes/kubernetes.io~projected/kube-api-access-hj6jk major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81258c01-d653-4a05-87ca-8373e7fad83d/volumes/kubernetes.io~projected/kube-api-access-b74st:{mountpoint:/var/lib/kubelet/pods/81258c01-d653-4a05-87ca-8373e7fad83d/volumes/kubernetes.io~projected/kube-api-access-b74st major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81258c01-d653-4a05-87ca-8373e7fad83d/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/81258c01-d653-4a05-87ca-8373e7fad83d/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/994c018e-e6c5-4d21-adac-9540676a67c0/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/994c018e-e6c5-4d21-adac-9540676a67c0/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/994c018e-e6c5-4d21-adac-9540676a67c0/volumes/kubernetes.io~projected/kube-api-access-9r7kr:{mountpoint:/var/lib/kubelet/pods/994c018e-e6c5-4d21-adac-9540676a67c0/volumes/kubernetes.io~projected/kube-api-access-9r7kr major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/994c018e-e6c5-4d21-adac-9540676a67c0/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/994c018e-e6c5-4d21-adac-9540676a67c0/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/995884d0-cb39-4c07-9dbe-5cd3529d5908/volumes/kubernetes.io~projected/kube-api-access-fb9nw:{mountpoint:/var/lib/kubelet/pods/995884d0-cb39-4c07-9dbe-5cd3529d5908/volumes/kubernetes.io~projected/kube-api-access-fb9nw major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0661ca3-6f0a-4744-9d50-f2baa1a80b45/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/b0661ca3-6f0a-4744-9d50-f2baa1a80b45/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3f55f18-4143-4806-b9e4-989e153f7e38/volumes/kubernetes.io~projected/kube-api-access-m4gkr:{mountpoint:/var/lib/kubelet/pods/b3f55f18-4143-4806-b9e4-989e153f7e38/volumes/kubernetes.io~projected/kube-api-access-m4gkr major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da571014-afa3-4a40-a083-dc708bf83ba3/volumes/kubernetes.io~projected/kube-api-access-9rxlb:{mountpoint:/var/lib/kubelet/pods/da571014-afa3-4a40-a083-dc708bf83ba3/volumes/kubernetes.io~projected/kube-api-access-9rxlb major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~empty-dir/config-out major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~projected/kube-api-access-mghhs:{mountpoint:/var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~projected/kube-api-access-mghhs major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~projected/tls-assets major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~secret/config major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~secret/web-config major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0/volumes/kubernetes.io~empty-dir/tmp major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0/volumes/kubernetes.io~projected/kube-api-access-ndpj2:{mountpoint:/var/lib/kubelet/pods/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0/volumes/kubernetes.io~projected/kube-api-access-ndpj2 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcf5056b-ef4e-430b-8b5c-83fad65af440/volumes/kubernetes.io~projected/kube-api-access-kghwp:{mountpoint:/var/lib/kubelet/pods/fcf5056b-ef4e-430b-8b5c-83fad65af440/volumes/kubernetes.io~projected/kube-api-access-kghwp major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcf5056b-ef4e-430b-8b5c-83fad65af440/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/fcf5056b-ef4e-430b-8b5c-83fad65af440/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcf5056b-ef4e-430b-8b5c-83fad65af440/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/fcf5056b-ef4e-430b-8b5c-83fad65af440/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffa2afe9-832d-452f-855d-982f9e47a601/volumes/kubernetes.io~projected/kube-api-access-8mplm:{mountpoint:/var/lib/kubelet/pods/ffa2afe9-832d-452f-855d-982f9e47a601/volumes/kubernetes.io~projected/kube-api-access-8mplm major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffa2afe9-832d-452f-855d-982f9e47a601/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/ffa2afe9-832d-452f-855d-982f9e47a601/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffa2afe9-832d-452f-855d-982f9e47a601/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/ffa2afe9-832d-452f-855d-982f9e47a601/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:249 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/80750b0edd56376fe23d54a7c2fb0cc3c90fb5174c962a9c2bc9dcae020f17ea/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/1507d87a5c5506e245bd25627fde558647ae05fc6f995638850d63d47b8f26a8/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/239a75389f44480e4094ca7d6d32e0450c59dbfd7ea38683920d9cf1feba3cdb/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/523f246a9064a6110694e19db693bf400f2021c8d9aeda2a8bbbcd378f778791/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/79596609c24f333d5bc5c804d876eedf538fa47ee5ef854ada28f3040c1d2279/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/847f091c6246694c9007ca028d4812bbf8f7ed30fb927733f4cd8078e824176e/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/8c0de77b99f8a2fd8c30f742653905ac6ddc696f6ca461f36311ed3f5a85e016/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/5a59c5a03c82697186f6f199ec85bd1e8a106e78924388cac9643f95eb4ae6d0/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/5a9c7c18fd610c827fa0ea3a631a28a293af1ed95e99e6d3686349d8a3ffd958/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/760f77844fbbb9a24a5d7c2a6519e4c98f7791399d807b8d33157de4915cce71/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/72d24b332a43efd71fa1956ed53102d5892e77f0aaa92b66ca2beb9304443312/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-150:{mountpoint:/var/lib/containers/storage/overlay/2f2465e2add4eb7443fe009909ffd4a6c998cbba9f6ecc3049346aa4747178f9/merged major:0 minor:150 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/28f950ef8ac968d8a231a3e50a071036ad0b4e328a839fc739e8244ad6732747/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/25ea4472a1e5582a6e1a1647d9c5cf18cf84adb8d09c278345f56df33d96d851/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/da174d8c6abb3db4b3bb4d9567e6c034678e4796788deef7320cd0f1a9f6e77a/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/a04bc29e2c4935edcfb254e73604a53b83d896252acbb23fd452f94ee5df31e9/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/6ec6991f584478005f17757dcc42722b1cd794abf1548dddb06af089bbd7a1e8/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/3ea7a500ed7e26b3379ed845aeb3427ffcab82bd40c1ef1e9d6faba05df4730a/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/dcdfd02c2fd50d8820193f07aa2e660dbe554fdc8358d7dd870fb7c8a0e8dba5/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/06bf9a9bde6fe88156a0f67c5a1bfecaff1a49d558c2ab9f9c6e119fdb635b92/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/ca529571771fec1baa5e329a30a9dc3ff0df6e1fbec4d1ec5425549c19c2d1eb/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/c178cd862c2ab64a65d4363eb24a1258fb21d04411832998729e656ada95b05c/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/17ad195bf5ea6826aba8454c262f6e93f507830b6a044ea3e0f08195df593faa/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/50b04aa44e7c5621d33b87d93ce211ef8286627450417aa2d1069a2155fb6540/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/c83710d50004dbf17f225fc83a02c67f129725fdf02d0bd85fb26ae78f53a768/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/e6b388eae28eee9c280f3ed4705bbc47de9f42030867ec1a9bf4737787fdc0bb/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/f790a4c996859b1d2ae4ca349be29aed29ba39ef292fd77393df7b487e6785a2/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/5878d5fadea6bedc7e0750f3682572a84df9a60a5d69ae946894cba97a1b9fa0/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/a07e5c84c40c5e6f9aadfd19636fd57af86b792ef7a87fa27779c55cf0a72b94/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/9077f0dabbab6eaa25345c86d6f2b46e0c75d423751dc9975df5399795ded22c/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/11f78832bdf76ecd7f94b91abd0b3a36fde91302d070b62c1755fcfb293015c9/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/5db5de0b034f75f968c12689f1dd6f02910c1c3c78a97ca3eefdd9a8cde436de/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/d01b7e8098e92139bbd9f1725b5588d91b3009aa8e86dda295e891ce8dfc0c75/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/2d286df1e9419b1d40e2623635a38224e5ea91113918a3ece3418ee47b8e1414/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/5ae3f4217278a54348aa36b369807ce0367aae65f63c54d724ec317dc2503fc6/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/b9fa891c2cc4ef272462513346147bc823c87b68b0606873bf1eef5a2fd1fe19/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/4d8523f024f5cb530a13a604c84f9082f7275e93411076690d159d26e28fc185/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/95010d828873baf572e7c65cdda0333a3991726bbf0de95c783a851fcd448f2e/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/7977a947ae11e95358bd2e229449b83640f00f4f9dba7d2fda4424131354c27f/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/7f2280a849b033a4d5d2d100f361dd872f82d8d44432d1c28755ca18f7232bf2/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/5c3344c182160b049028f72927ee2b584fa95d028c867a46b9bee951fbdb4547/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/3391204926b56c7c7e66f813010c0a13a255411b74de3e51da4e19b8964ed9e7/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/77945929d44fc23bb2622787f17e8886f1b06538b0173013e676895f055ef512/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/3b5e6b985321bbb3b02ef5c877ab7fc1dfe1f94db192cecc744d46344fb78a1a/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/1f911b52c021b24c86efc0b747697f3b6e056ab507fb0b1078141c5b175f14b9/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/d2e6796710d96662437d85bebeb483ddbdc958a5be6a21f26660112611bf836d/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/b7c6781edb0282fcd030bb490470a65785639f3c7ea101fc9ad3730efb63afe8/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/7595e2bec06e7058a690ca6d66e9e02fc1348b1723ff28c9a15bc4439d3d1f54/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/7b2a4436cb4963611082dd3e9bde37583bfe68ef8b1c4c45ecbd3a6b53af6696/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/818cddd387a4aa410dd9cad04ba58ec5320d749799883daba20a3562c7ca34ae/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-62:{mountpoint:/var/lib/containers/storage/overlay/ce6c643bcb2612f6b61ffe872f38fc0dd737fdf214248e177ce81e666357c4c5/merged major:0 minor:62 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/11cc18856d31feff725df0ba125f8f5cb6dade25a5b61ce60f68761e7485aab7/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/98959e8febe2b23d14c7e95bdc836de2337a24ca6c7a06ef16b37320c7c378d8/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/a6ce7fb6a0523b564eefa5040ad21ca139e8f4adece7c4e640a7556fe4c1a9ea/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/264efb8fd493ed6fb399ec669a75b9f9d65e31a720a647dd7d9774847113f076/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/98ff6f260a8d1f9d2482ff8383c051381620288845d50c94f1654b50d083b277/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/fc71e294ce6210786da0310924fca229d0221926adf853cdaf70ba53b02b022f/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/6a365101ab5c7582fb57fc5a0e2bb7d0e4893dc8617099fb8dd6332dc1c0e881/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/e5dcad095d645eefebc05cc8658b43ff9aa599f36290dfe814c1e841ed95d5b8/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/9ea184d7d84394e0db7a291a0286f178db06975030f75c74ef2041739bd450a0/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 00:43:19.755235 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.754365 5300 manager.go:217] Machine: {Timestamp:2026-10-30 00:43:19.753702421 +0000 UTC m=+0.069570755 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3190901 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20b867d17fe0b98927a8b0bdf4f4e8 SystemUUID:ec20b867-d17f-e0b9-8927-a8b0bdf4f4e8 BootID:e6864bfd-24c6-4313-986b-17c0c493cdb0 Filesystems:[{Device:/var/lib/kubelet/pods/1f1c6441-4ebb-48d6-9e09-bf69e431eb57/volumes/kubernetes.io~projected/kube-api-access-4l7h2 DeviceMajor:0 DeviceMinor:56 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01c112ff76552443b9e39c5b79208ec96ef2aa7f3e6b7573a69ed107b9f71355/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fcf5056b-ef4e-430b-8b5c-83fad65af440/volumes/kubernetes.io~projected/kube-api-access-kghwp DeviceMajor:0 DeviceMinor:283 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:329 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b22701a-4358-4fd4-92d9-bd838f5af629/volumes/kubernetes.io~projected/kube-api-access-b9srn DeviceMajor:0 DeviceMinor:202 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:332 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:333 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/641e3ef2e21a72d92a5de7c37c8369f43fc34067da733bababa0b8bba76242c1/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:324 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0/volumes/kubernetes.io~projected/kube-api-access-ndpj2 DeviceMajor:0 DeviceMinor:55 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c963d0df141bd6ca560142b1b23b521cdf62b24164a3baab88f5318e8503bffd/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3c78afce-0bbb-49e1-aceb-ae4a69844075/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:229 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0661ca3-6f0a-4744-9d50-f2baa1a80b45/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:304 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:328 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/49f04f3b-49d5-4c22-88a2-e33e106853ce/volumes/kubernetes.io~projected/kube-api-access-tlnvv DeviceMajor:0 DeviceMinor:51 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/839ff87247ef7ca34d8a7fecffcd2d142d915609c291cc4c257adcda2241eb00/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:323 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:325 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1c2f1e93-f19e-4c0f-944e-7f52261f6e81/volumes/kubernetes.io~projected/kube-api-access-9x2hb DeviceMajor:0 DeviceMinor:48 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e23232e117081451795faf1fc0b232b8ae80acdaa22d5d0a82c2d92146c78453/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/994c018e-e6c5-4d21-adac-9540676a67c0/volumes/kubernetes.io~projected/kube-api-access-9r7kr DeviceMajor:0 DeviceMinor:50 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6037462f1e78bb1f94b06d5412aa46efec347619ee0909ac5800f8efa81d3f45/userdata/shm DeviceMajor:0 DeviceMinor:309 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:324 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/994c018e-e6c5-4d21-adac-9540676a67c0/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fad95519ff0692168a93889a183f5b95dbe2f7c1378dd82b2fa81af27995108d/userdata/shm DeviceMajor:0 DeviceMinor:306 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a39b78a775f0f6b5a51dd6831da2352abac4740bed387f3af7c58b4c2ef8676c/userdata/shm DeviceMajor:0 DeviceMinor:344 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7127ca6b-3962-47a9-9089-07bcba277bc7/volumes/kubernetes.io~projected/kube-api-access-hj6jk DeviceMajor:0 DeviceMinor:57 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/da571014-afa3-4a40-a083-dc708bf83ba3/volumes/kubernetes.io~projected/kube-api-access-9rxlb DeviceMajor:0 DeviceMinor:58 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81258c01-d653-4a05-87ca-8373e7fad83d/volumes/kubernetes.io~projected/kube-api-access-b74st DeviceMajor:0 DeviceMinor:203 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:52 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/653fb098-e2a1-4674-9baa-6975e9dd0fec/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/67bde315-b2ea-4d17-88d8-f290cd2c994e/volumes/kubernetes.io~projected/kube-api-access-mp68j DeviceMajor:0 DeviceMinor:299 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ef8f37acadd6669c5af3620533b330d8f1b32b11361b56756eadaf7258e388f/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~projected/kube-api-access-mghhs DeviceMajor:0 DeviceMinor:334 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5dff88d3-2e35-40e9-8aea-208519485adc/volumes/kubernetes.io~projected/kube-api-access-w48mv DeviceMajor:0 DeviceMinor:59 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:330 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc6e866c85861f00cee7ad73ec83efa452092d9d609585d1e4be909d96416e2c/userdata/shm DeviceMajor:0 DeviceMinor:337 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3c78afce-0bbb-49e1-aceb-ae4a69844075/volumes/kubernetes.io~projected/kube-api-access-bwqqr DeviceMajor:0 DeviceMinor:230 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b3f55f18-4143-4806-b9e4-989e153f7e38/volumes/kubernetes.io~projected/kube-api-access-m4gkr DeviceMajor:0 DeviceMinor:308 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d9783f5758a10e5b89b5103e04b1261b0cfd92b52c5cf73c558c6b763e53a481/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/653fb098-e2a1-4674-9baa-6975e9dd0fec/volumes/kubernetes.io~projected/kube-api-access-s8kj9 DeviceMajor:0 DeviceMinor:204 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fcf5056b-ef4e-430b-8b5c-83fad65af440/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:277 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8aedcc0431068aad1e5868f70d8904e299ca29410447c75871583ef000efb149/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b22701a-4358-4fd4-92d9-bd838f5af629/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab4805c524196e761de616228d521027344068919a2c839e780a27bb20f5276c/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/07e33ac0-f034-477b-8e5a-aaf82a23d7fa/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:341 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffa2afe9-832d-452f-855d-982f9e47a601/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:248 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/557a69cba7345b9287acd09ba0cbdc66d6c50244e6730aa9908eab1505bd1617/userdata/shm DeviceMajor:0 DeviceMinor:313 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4cd9efa42b7843ae35ae9d8eadc51084370007512f89c38570b4c4ecc65a59f2/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/81258c01-d653-4a05-87ca-8373e7fad83d/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1c2f1e93-f19e-4c0f-944e-7f52261f6e81/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:305 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/ffa2afe9-832d-452f-855d-982f9e47a601/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:249 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:326 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/67bde315-b2ea-4d17-88d8-f290cd2c994e/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:295 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/07e33ac0-f034-477b-8e5a-aaf82a23d7fa/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:342 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-150 DeviceMajor:0 DeviceMinor:150 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b3abef169ea86e60b9acf2249c3cb82e1023cf6d9e242acaa4994c5af54e583/userdata/shm DeviceMajor:0 DeviceMinor:250 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3494cae29415aae77bc7ed41918f4dc56688122704debb112eb50231cc64e9ba/userdata/shm DeviceMajor:0 DeviceMinor:284 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:53 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1a21ea02e80d4f6d786dbd8b2d11fb89a274e7c3ff0c3b317588934510b97e06/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/874ddcd3a5bb7eb513a1939007b192044d8483cb5e1d5b6603821d7adfc204e6/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/004ba09d25f8f7a4344e34113018b282226d51e97b12c38e6fc6cf1d24b021f8/userdata/shm DeviceMajor:0 DeviceMinor:300 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:331 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/07e33ac0-f034-477b-8e5a-aaf82a23d7fa/volumes/kubernetes.io~projected/kube-api-access-2726g DeviceMajor:0 DeviceMinor:343 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/995884d0-cb39-4c07-9dbe-5cd3529d5908/volumes/kubernetes.io~projected/kube-api-access-fb9nw DeviceMajor:0 DeviceMinor:49 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b76808fc47c412c7ba94d1e61361c17514fc485ba206f74e99d4d3584f101901/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3c78afce-0bbb-49e1-aceb-ae4a69844075/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:225 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ffa2afe9-832d-452f-855d-982f9e47a601/volumes/kubernetes.io~projected/kube-api-access-8mplm DeviceMajor:0 DeviceMinor:251 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:327 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-62 DeviceMajor:0 DeviceMinor:62 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cd71fcdd9fcb74d62ca6fd43f9446fd30908000a64453e19d6f2e8de7bafcadd/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/994c018e-e6c5-4d21-adac-9540676a67c0/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/271e26db-6a8b-4e46-afb7-139cbc57c1e1/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:54 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fcf5056b-ef4e-430b-8b5c-83fad65af440/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:282 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:004ba09d25f8f7a MacAddress:2e:9d:72:d7:95:9c Speed:10000 Mtu:8901} {Name:01c112ff7655244 MacAddress:46:5a:23:91:96:20 Speed:10000 Mtu:8901} {Name:557a69cba7345b9 MacAddress:2e:74:5a:46:d9:71 Speed:10000 Mtu:8901} {Name:6037462f1e78bb1 MacAddress:f2:0b:19:15:46:19 Speed:10000 Mtu:8901} {Name:874ddcd3a5bb7eb MacAddress:9a:7c:42:7d:e3:7c Speed:10000 Mtu:8901} {Name:9b3abef169ea86e MacAddress:c2:93:e4:13:a5:90 Speed:10000 Mtu:8901} {Name:a39b78a775f0f6b MacAddress:a6:e4:dc:4b:3d:2a Speed:10000 Mtu:8901} {Name:ab4805c524196e7 MacAddress:16:b9:ca:da:23:c4 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e3:d0:ff:f3 Speed:0 Mtu:9001} {Name:br-int MacAddress:fa:fc:4e:80:52:f1 Speed:0 Mtu:8901} {Name:cc6e866c85861f0 MacAddress:7e:e7:56:d5:4a:9d Speed:10000 Mtu:8901} {Name:e23232e11708145 MacAddress:4e:cf:d5:9e:9e:b0 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e3:d0:ff:f3 Speed:0 Mtu:9001} {Name:fad95519ff06921 MacAddress:4a:c9:0f:c7:5e:94 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:7a:16:ac:6c:5a:bb Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:7e:12:42:41:b9:6e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 00:43:19.755235 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.754999 5300 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 00:43:19.755235 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.755062 5300 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 00:43:19.755464 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.755369 5300 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 00:43:19.755536 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.755394 5300 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-206.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 00:43:19.755577 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.755537 5300 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 00:43:19.755577 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.755545 5300 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 00:43:19.755577 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.755559 5300 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:43:19.755659 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.755582 5300 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 00:43:19.755687 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.755676 5300 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 00:43:19.755877 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.755866 5300 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 00:43:19.755989 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.755980 5300 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 00:43:19.756023 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.756017 5300 kubelet.go:493] "Attempting to sync node with API server" Oct 30 00:43:19.756051 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.756027 5300 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 00:43:19.756051 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.756037 5300 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 00:43:19.756051 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.756047 5300 kubelet.go:405] "Adding apiserver pod source" Oct 30 00:43:19.756135 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.756062 5300 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 00:43:19.756881 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.756864 5300 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 00:43:19.756940 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.756882 5300 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:43:19.761388 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.761314 5300 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 00:43:19.773427 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.773404 5300 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 00:43:19.773427 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.773422 5300 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 00:43:19.773569 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.773533 5300 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 00:43:19.773749 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.773737 5300 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 00:43:19.773810 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.773792 5300 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 00:43:19.773943 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.773933 5300 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 00:43:19.773973 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.773950 5300 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 00:43:19.773973 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.773957 5300 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 00:43:19.773973 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.773963 5300 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 00:43:19.773973 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.773969 5300 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 00:43:19.774073 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.773977 5300 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 00:43:19.774073 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.773987 5300 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 00:43:19.774073 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.773995 5300 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 00:43:19.774073 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.774002 5300 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 00:43:19.774073 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.774009 5300 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 00:43:19.774073 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.774018 5300 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 00:43:19.774073 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.774027 5300 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 00:43:19.774073 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.774054 5300 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 00:43:19.774073 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.774064 5300 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 00:43:19.777694 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.777676 5300 server.go:1263] "Started kubelet" Oct 30 00:43:19.777822 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.777727 5300 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 00:43:19.780270 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.777851 5300 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 00:43:19.780270 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.777949 5300 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 00:43:19.780270 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.778261 5300 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 00:43:19.780270 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.779113 5300 server.go:317] "Adding debug handlers to kubelet server" Oct 30 00:43:19.778249 ip-10-0-140-206 systemd[1]: Started Kubernetes Kubelet. Oct 30 00:43:19.781660 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.781635 5300 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 00:43:19.781741 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.781657 5300 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 00:43:19.781741 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.781716 5300 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:37:10 +0000 UTC" deadline="2028-04-18 09:17:09.690952133 +0000 UTC" Oct 30 00:43:19.781859 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.781750 5300 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12872h33m49.909207425s" Oct 30 00:43:19.781859 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.781794 5300 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 00:43:19.781859 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.781809 5300 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 00:43:19.782008 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.781914 5300 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 00:43:19.782594 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:43:19.782559 5300 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-206.ec2.internal\" not found" Oct 30 00:43:19.784925 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.784897 5300 factory.go:55] Registering systemd factory Oct 30 00:43:19.784925 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.784926 5300 factory.go:223] Registration of the systemd container factory successfully Oct 30 00:43:19.787787 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.786442 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c78afce-0bbb-49e1-aceb-ae4a69844075" volumeName="kubernetes.io/secret/3c78afce-0bbb-49e1-aceb-ae4a69844075-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 00:43:19.787787 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.786571 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f04f3b-49d5-4c22-88a2-e33e106853ce" volumeName="kubernetes.io/configmap/49f04f3b-49d5-4c22-88a2-e33e106853ce-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 00:43:19.787787 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.786666 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67bde315-b2ea-4d17-88d8-f290cd2c994e" volumeName="kubernetes.io/secret/67bde315-b2ea-4d17-88d8-f290cd2c994e-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 00:43:19.787787 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.786872 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e991db07-d748-4ced-8d7a-4266194e89d8" volumeName="kubernetes.io/empty-dir/e991db07-d748-4ced-8d7a-4266194e89d8-config-out" seLinuxMountContext="" Oct 30 00:43:19.787787 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.786908 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffa2afe9-832d-452f-855d-982f9e47a601" volumeName="kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-service-ca" seLinuxMountContext="" Oct 30 00:43:19.787787 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.786927 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07e33ac0-f034-477b-8e5a-aaf82a23d7fa" volumeName="kubernetes.io/empty-dir/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-tmp" seLinuxMountContext="" Oct 30 00:43:19.787787 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.786981 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f1c6441-4ebb-48d6-9e09-bf69e431eb57" volumeName="kubernetes.io/projected/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-kube-api-access-4l7h2" seLinuxMountContext="" Oct 30 00:43:19.787787 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.787112 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c78afce-0bbb-49e1-aceb-ae4a69844075" volumeName="kubernetes.io/configmap/3c78afce-0bbb-49e1-aceb-ae4a69844075-metrics-client-ca" seLinuxMountContext="" Oct 30 00:43:19.787787 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.787138 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="994c018e-e6c5-4d21-adac-9540676a67c0" volumeName="kubernetes.io/configmap/994c018e-e6c5-4d21-adac-9540676a67c0-env-overrides" seLinuxMountContext="" Oct 30 00:43:19.787787 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.787197 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e991db07-d748-4ced-8d7a-4266194e89d8" volumeName="kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 00:43:19.787787 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.787326 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e991db07-d748-4ced-8d7a-4266194e89d8" volumeName="kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 00:43:19.787787 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.787377 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcf5056b-ef4e-430b-8b5c-83fad65af440" volumeName="kubernetes.io/configmap/fcf5056b-ef4e-430b-8b5c-83fad65af440-metrics-client-ca" seLinuxMountContext="" Oct 30 00:43:19.787787 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.787467 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffa2afe9-832d-452f-855d-982f9e47a601" volumeName="kubernetes.io/secret/ffa2afe9-832d-452f-855d-982f9e47a601-console-serving-cert" seLinuxMountContext="" Oct 30 00:43:19.787787 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.787604 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="653fb098-e2a1-4674-9baa-6975e9dd0fec" volumeName="kubernetes.io/secret/653fb098-e2a1-4674-9baa-6975e9dd0fec-cert" seLinuxMountContext="" Oct 30 00:43:19.787787 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.787669 5300 factory.go:153] Registering CRI-O factory Oct 30 00:43:19.787787 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.787694 5300 factory.go:223] Registration of the crio container factory successfully Oct 30 00:43:19.787787 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.787754 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da571014-afa3-4a40-a083-dc708bf83ba3" volumeName="kubernetes.io/projected/da571014-afa3-4a40-a083-dc708bf83ba3-kube-api-access-9rxlb" seLinuxMountContext="" Oct 30 00:43:19.788557 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.787896 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e991db07-d748-4ced-8d7a-4266194e89d8" volumeName="kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 00:43:19.788557 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.787804 5300 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 00:43:19.788557 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.787954 5300 factory.go:103] Registering Raw factory Oct 30 00:43:19.788557 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.787974 5300 manager.go:1199] Started watching for new ooms in manager Oct 30 00:43:19.788557 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.788012 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e991db07-d748-4ced-8d7a-4266194e89d8" volumeName="kubernetes.io/projected/e991db07-d748-4ced-8d7a-4266194e89d8-tls-assets" seLinuxMountContext="" Oct 30 00:43:19.788557 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.788029 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffa2afe9-832d-452f-855d-982f9e47a601" volumeName="kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-oauth-serving-cert" seLinuxMountContext="" Oct 30 00:43:19.788557 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.788199 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da571014-afa3-4a40-a083-dc708bf83ba3" volumeName="kubernetes.io/configmap/da571014-afa3-4a40-a083-dc708bf83ba3-cni-binary-copy" seLinuxMountContext="" Oct 30 00:43:19.788889 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.788841 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e991db07-d748-4ced-8d7a-4266194e89d8" volumeName="kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 00:43:19.788984 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.788965 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07e33ac0-f034-477b-8e5a-aaf82a23d7fa" volumeName="kubernetes.io/secret/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-oauth-config" seLinuxMountContext="" Oct 30 00:43:19.789035 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.788989 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="271e26db-6a8b-4e46-afb7-139cbc57c1e1" volumeName="kubernetes.io/secret/271e26db-6a8b-4e46-afb7-139cbc57c1e1-agent-certs" seLinuxMountContext="" Oct 30 00:43:19.789035 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.789012 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81258c01-d653-4a05-87ca-8373e7fad83d" volumeName="kubernetes.io/empty-dir/81258c01-d653-4a05-87ca-8373e7fad83d-tmp-dir" seLinuxMountContext="" Oct 30 00:43:19.789035 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.789028 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81258c01-d653-4a05-87ca-8373e7fad83d" volumeName="kubernetes.io/secret/81258c01-d653-4a05-87ca-8373e7fad83d-metrics-tls" seLinuxMountContext="" Oct 30 00:43:19.789160 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.789047 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e991db07-d748-4ced-8d7a-4266194e89d8" volumeName="kubernetes.io/empty-dir/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-db" seLinuxMountContext="" Oct 30 00:43:19.789160 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.789066 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" volumeName="kubernetes.io/projected/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-kube-api-access-9x2hb" seLinuxMountContext="" Oct 30 00:43:19.789160 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.789082 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5dff88d3-2e35-40e9-8aea-208519485adc" volumeName="kubernetes.io/projected/5dff88d3-2e35-40e9-8aea-208519485adc-kube-api-access-w48mv" seLinuxMountContext="" Oct 30 00:43:19.789289 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.789269 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e991db07-d748-4ced-8d7a-4266194e89d8" volumeName="kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 00:43:19.789338 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.789196 5300 manager.go:319] Starting recovery of all containers Oct 30 00:43:19.789338 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.789313 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffa2afe9-832d-452f-855d-982f9e47a601" volumeName="kubernetes.io/secret/ffa2afe9-832d-452f-855d-982f9e47a601-console-oauth-config" seLinuxMountContext="" Oct 30 00:43:19.789338 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.789327 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07e33ac0-f034-477b-8e5a-aaf82a23d7fa" volumeName="kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-trusted-ca-bundle" seLinuxMountContext="" Oct 30 00:43:19.789507 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.789445 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07e33ac0-f034-477b-8e5a-aaf82a23d7fa" volumeName="kubernetes.io/projected/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-kube-api-access-2726g" seLinuxMountContext="" Oct 30 00:43:19.790007 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.789961 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7127ca6b-3962-47a9-9089-07bcba277bc7" volumeName="kubernetes.io/projected/7127ca6b-3962-47a9-9089-07bcba277bc7-kube-api-access-hj6jk" seLinuxMountContext="" Oct 30 00:43:19.790007 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.789994 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffa2afe9-832d-452f-855d-982f9e47a601" volumeName="kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-console-config" seLinuxMountContext="" Oct 30 00:43:19.790007 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790008 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07e33ac0-f034-477b-8e5a-aaf82a23d7fa" volumeName="kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-oauth-serving-cert" seLinuxMountContext="" Oct 30 00:43:19.790168 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790020 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07e33ac0-f034-477b-8e5a-aaf82a23d7fa" volumeName="kubernetes.io/secret/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-serving-cert" seLinuxMountContext="" Oct 30 00:43:19.790168 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790032 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c2f1e93-f19e-4c0f-944e-7f52261f6e81" volumeName="kubernetes.io/secret/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-metrics-certs" seLinuxMountContext="" Oct 30 00:43:19.790168 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790044 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f04f3b-49d5-4c22-88a2-e33e106853ce" volumeName="kubernetes.io/configmap/49f04f3b-49d5-4c22-88a2-e33e106853ce-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 00:43:19.790168 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790057 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81258c01-d653-4a05-87ca-8373e7fad83d" volumeName="kubernetes.io/projected/81258c01-d653-4a05-87ca-8373e7fad83d-kube-api-access-b74st" seLinuxMountContext="" Oct 30 00:43:19.790168 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790070 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da571014-afa3-4a40-a083-dc708bf83ba3" volumeName="kubernetes.io/configmap/da571014-afa3-4a40-a083-dc708bf83ba3-multus-daemon-config" seLinuxMountContext="" Oct 30 00:43:19.790168 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790089 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e991db07-d748-4ced-8d7a-4266194e89d8" volumeName="kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 00:43:19.790168 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790104 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e991db07-d748-4ced-8d7a-4266194e89d8" volumeName="kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-config" seLinuxMountContext="" Oct 30 00:43:19.790168 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790116 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67bde315-b2ea-4d17-88d8-f290cd2c994e" volumeName="kubernetes.io/projected/67bde315-b2ea-4d17-88d8-f290cd2c994e-kube-api-access-mp68j" seLinuxMountContext="" Oct 30 00:43:19.790168 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790127 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="994c018e-e6c5-4d21-adac-9540676a67c0" volumeName="kubernetes.io/projected/994c018e-e6c5-4d21-adac-9540676a67c0-kube-api-access-9r7kr" seLinuxMountContext="" Oct 30 00:43:19.790168 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790138 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="995884d0-cb39-4c07-9dbe-5cd3529d5908" volumeName="kubernetes.io/empty-dir/995884d0-cb39-4c07-9dbe-5cd3529d5908-tmp-dir" seLinuxMountContext="" Oct 30 00:43:19.790168 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790151 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e991db07-d748-4ced-8d7a-4266194e89d8" volumeName="kubernetes.io/projected/e991db07-d748-4ced-8d7a-4266194e89d8-kube-api-access-mghhs" seLinuxMountContext="" Oct 30 00:43:19.790168 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790162 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b22701a-4358-4fd4-92d9-bd838f5af629" volumeName="kubernetes.io/configmap/0b22701a-4358-4fd4-92d9-bd838f5af629-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 00:43:19.790168 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790175 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7127ca6b-3962-47a9-9089-07bcba277bc7" volumeName="kubernetes.io/configmap/7127ca6b-3962-47a9-9089-07bcba277bc7-iptables-alerter-script" seLinuxMountContext="" Oct 30 00:43:19.790793 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790187 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e991db07-d748-4ced-8d7a-4266194e89d8" volumeName="kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 00:43:19.790793 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790198 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e991db07-d748-4ced-8d7a-4266194e89d8" volumeName="kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-grpc-tls" seLinuxMountContext="" Oct 30 00:43:19.790793 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790211 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5dff88d3-2e35-40e9-8aea-208519485adc" volumeName="kubernetes.io/configmap/5dff88d3-2e35-40e9-8aea-208519485adc-serviceca" seLinuxMountContext="" Oct 30 00:43:19.790793 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790222 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="653fb098-e2a1-4674-9baa-6975e9dd0fec" volumeName="kubernetes.io/projected/653fb098-e2a1-4674-9baa-6975e9dd0fec-kube-api-access-s8kj9" seLinuxMountContext="" Oct 30 00:43:19.790793 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790230 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e991db07-d748-4ced-8d7a-4266194e89d8" volumeName="kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 00:43:19.790793 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790239 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e991db07-d748-4ced-8d7a-4266194e89d8" volumeName="kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 00:43:19.790793 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790249 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcf5056b-ef4e-430b-8b5c-83fad65af440" volumeName="kubernetes.io/configmap/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 00:43:19.790793 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790257 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcf5056b-ef4e-430b-8b5c-83fad65af440" volumeName="kubernetes.io/projected/fcf5056b-ef4e-430b-8b5c-83fad65af440-kube-api-access-kghwp" seLinuxMountContext="" Oct 30 00:43:19.790793 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790267 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e991db07-d748-4ced-8d7a-4266194e89d8" volumeName="kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 00:43:19.790793 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790283 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b22701a-4358-4fd4-92d9-bd838f5af629" volumeName="kubernetes.io/empty-dir/0b22701a-4358-4fd4-92d9-bd838f5af629-data-volume" seLinuxMountContext="" Oct 30 00:43:19.790793 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790291 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c78afce-0bbb-49e1-aceb-ae4a69844075" volumeName="kubernetes.io/projected/3c78afce-0bbb-49e1-aceb-ae4a69844075-kube-api-access-bwqqr" seLinuxMountContext="" Oct 30 00:43:19.790793 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790300 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f04f3b-49d5-4c22-88a2-e33e106853ce" volumeName="kubernetes.io/projected/49f04f3b-49d5-4c22-88a2-e33e106853ce-kube-api-access-tlnvv" seLinuxMountContext="" Oct 30 00:43:19.790793 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790308 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="994c018e-e6c5-4d21-adac-9540676a67c0" volumeName="kubernetes.io/configmap/994c018e-e6c5-4d21-adac-9540676a67c0-ovnkube-script-lib" seLinuxMountContext="" Oct 30 00:43:19.790793 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790315 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="994c018e-e6c5-4d21-adac-9540676a67c0" volumeName="kubernetes.io/secret/994c018e-e6c5-4d21-adac-9540676a67c0-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 00:43:19.790793 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790324 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcf5056b-ef4e-430b-8b5c-83fad65af440" volumeName="kubernetes.io/secret/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-tls" seLinuxMountContext="" Oct 30 00:43:19.790793 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790333 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffa2afe9-832d-452f-855d-982f9e47a601" volumeName="kubernetes.io/projected/ffa2afe9-832d-452f-855d-982f9e47a601-kube-api-access-8mplm" seLinuxMountContext="" Oct 30 00:43:19.790793 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790342 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="995884d0-cb39-4c07-9dbe-5cd3529d5908" volumeName="kubernetes.io/projected/995884d0-cb39-4c07-9dbe-5cd3529d5908-kube-api-access-fb9nw" seLinuxMountContext="" Oct 30 00:43:19.790793 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790455 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07e33ac0-f034-477b-8e5a-aaf82a23d7fa" volumeName="kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-config" seLinuxMountContext="" Oct 30 00:43:19.790793 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790468 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b22701a-4358-4fd4-92d9-bd838f5af629" volumeName="kubernetes.io/projected/0b22701a-4358-4fd4-92d9-bd838f5af629-kube-api-access-b9srn" seLinuxMountContext="" Oct 30 00:43:19.790793 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790481 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f04f3b-49d5-4c22-88a2-e33e106853ce" volumeName="kubernetes.io/configmap/49f04f3b-49d5-4c22-88a2-e33e106853ce-cni-binary-copy" seLinuxMountContext="" Oct 30 00:43:19.790793 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790492 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81258c01-d653-4a05-87ca-8373e7fad83d" volumeName="kubernetes.io/configmap/81258c01-d653-4a05-87ca-8373e7fad83d-config-volume" seLinuxMountContext="" Oct 30 00:43:19.790793 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790504 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0" volumeName="kubernetes.io/empty-dir/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-tuned" seLinuxMountContext="" Oct 30 00:43:19.790793 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790515 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcf5056b-ef4e-430b-8b5c-83fad65af440" volumeName="kubernetes.io/secret/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 00:43:19.790793 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790530 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3f55f18-4143-4806-b9e4-989e153f7e38" volumeName="kubernetes.io/projected/b3f55f18-4143-4806-b9e4-989e153f7e38-kube-api-access-m4gkr" seLinuxMountContext="" Oct 30 00:43:19.790793 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.790542 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="271e26db-6a8b-4e46-afb7-139cbc57c1e1" volumeName="kubernetes.io/configmap/271e26db-6a8b-4e46-afb7-139cbc57c1e1-konnectivity-ca" seLinuxMountContext="" Oct 30 00:43:19.791860 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.791203 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="994c018e-e6c5-4d21-adac-9540676a67c0" volumeName="kubernetes.io/configmap/994c018e-e6c5-4d21-adac-9540676a67c0-ovnkube-config" seLinuxMountContext="" Oct 30 00:43:19.791860 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.791231 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0661ca3-6f0a-4744-9d50-f2baa1a80b45" volumeName="kubernetes.io/secret/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-original-pull-secret" seLinuxMountContext="" Oct 30 00:43:19.791860 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.791242 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e991db07-d748-4ced-8d7a-4266194e89d8" volumeName="kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 00:43:19.791860 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.791254 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e991db07-d748-4ced-8d7a-4266194e89d8" volumeName="kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-web-config" seLinuxMountContext="" Oct 30 00:43:19.791860 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.791268 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0" volumeName="kubernetes.io/empty-dir/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-tmp" seLinuxMountContext="" Oct 30 00:43:19.791860 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.791282 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0" volumeName="kubernetes.io/projected/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-kube-api-access-ndpj2" seLinuxMountContext="" Oct 30 00:43:19.791860 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.791293 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b22701a-4358-4fd4-92d9-bd838f5af629" volumeName="kubernetes.io/secret/0b22701a-4358-4fd4-92d9-bd838f5af629-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 00:43:19.791860 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.791304 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c78afce-0bbb-49e1-aceb-ae4a69844075" volumeName="kubernetes.io/secret/3c78afce-0bbb-49e1-aceb-ae4a69844075-prometheus-operator-tls" seLinuxMountContext="" Oct 30 00:43:19.791860 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.791316 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e991db07-d748-4ced-8d7a-4266194e89d8" volumeName="kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 00:43:19.791860 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.791328 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e991db07-d748-4ced-8d7a-4266194e89d8" volumeName="kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 00:43:19.791860 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.791341 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcf5056b-ef4e-430b-8b5c-83fad65af440" volumeName="kubernetes.io/empty-dir/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-textfile" seLinuxMountContext="" Oct 30 00:43:19.791860 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.791353 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffa2afe9-832d-452f-855d-982f9e47a601" volumeName="kubernetes.io/empty-dir/ffa2afe9-832d-452f-855d-982f9e47a601-tmp" seLinuxMountContext="" Oct 30 00:43:19.791860 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.791365 5300 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07e33ac0-f034-477b-8e5a-aaf82a23d7fa" volumeName="kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-service-ca" seLinuxMountContext="" Oct 30 00:43:19.791860 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.791377 5300 reconstruct.go:97] "Volume reconstruction finished" Oct 30 00:43:19.791860 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.791385 5300 reconciler.go:29] "Reconciler: start to sync state" Oct 30 00:43:19.810750 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.810710 5300 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 00:43:19.812462 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.812436 5300 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 00:43:19.812462 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.812462 5300 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 00:43:19.812646 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.812490 5300 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 00:43:19.812646 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:43:19.812533 5300 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 00:43:19.820696 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.820644 5300 generic.go:356] "Generic (PLEG): container finished" podID="e991db07-d748-4ced-8d7a-4266194e89d8" containerID="4cde129fff51071d51c81973829d7b60cbdc27630f79782f7aa3fcb590cfe88f" exitCode=0 Oct 30 00:43:19.822050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.822027 5300 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:43:19.822162 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.822062 5300 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 00:43:19.823605 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.823586 5300 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:43:19.838553 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.838524 5300 generic.go:356] "Generic (PLEG): container finished" podID="fcf5056b-ef4e-430b-8b5c-83fad65af440" containerID="276deba1049dbbb6552b4db239bc59fa20bc594e454a7a3464eddd05f25602dd" exitCode=0 Oct 30 00:43:19.849525 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.849505 5300 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:43:19.849645 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.849569 5300 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 00:43:19.855987 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.855956 5300 generic.go:356] "Generic (PLEG): container finished" podID="49f04f3b-49d5-4c22-88a2-e33e106853ce" containerID="0972806bff466c119f69edd4bcf0fda3ccfab64fd1598916981aa71a67afd145" exitCode=0 Oct 30 00:43:19.855987 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.855986 5300 generic.go:356] "Generic (PLEG): container finished" podID="49f04f3b-49d5-4c22-88a2-e33e106853ce" containerID="59df6e494e5501738b2c9ce67ec9c0390dce9c1a45dcfeacd527ed40daf81ab7" exitCode=0 Oct 30 00:43:19.856155 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.855995 5300 generic.go:356] "Generic (PLEG): container finished" podID="49f04f3b-49d5-4c22-88a2-e33e106853ce" containerID="cfd77bb0128b9d21dac7a3b7f566b2576c0faecedada1216db4e1855f4a04e10" exitCode=0 Oct 30 00:43:19.856155 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.856005 5300 generic.go:356] "Generic (PLEG): container finished" podID="49f04f3b-49d5-4c22-88a2-e33e106853ce" containerID="0f9090f6f8f34f5af42dd21d7ec7a0422df95e56b90b425e1bb6960fd54ca22e" exitCode=0 Oct 30 00:43:19.856155 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.856014 5300 generic.go:356] "Generic (PLEG): container finished" podID="49f04f3b-49d5-4c22-88a2-e33e106853ce" containerID="2ddecffc4c5270c2aec387819c4e23016a87a2db107be8f7a9379739e339b1ac" exitCode=0 Oct 30 00:43:19.856155 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.856022 5300 generic.go:356] "Generic (PLEG): container finished" podID="49f04f3b-49d5-4c22-88a2-e33e106853ce" containerID="4a8d8c40faa74b5df1b39fedd3aeb654eb32844e06d0f7f1586a220b85b243a6" exitCode=0 Oct 30 00:43:19.865055 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.865029 5300 generic.go:356] "Generic (PLEG): container finished" podID="91d0355fca0e17295f697c41968d06d7" containerID="ffbf7c1727a78c296c0d9d7895a47b4281c10c91811181dee30c24386b8d574c" exitCode=0 Oct 30 00:43:19.884394 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.884339 5300 manager.go:324] Recovery completed Oct 30 00:43:19.903340 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.903321 5300 cpu_manager.go:226] "Starting" policy="none" Oct 30 00:43:19.903340 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.903338 5300 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 00:43:19.903460 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.903356 5300 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 00:43:19.903490 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.903470 5300 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 00:43:19.903530 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.903482 5300 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 00:43:19.903530 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.903499 5300 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 00:43:19.903530 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.903504 5300 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 00:43:19.903530 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.903509 5300 policy_none.go:50] "Start" Oct 30 00:43:19.903530 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.903517 5300 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 00:43:19.903530 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.903526 5300 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 00:43:19.904136 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.904120 5300 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 00:43:19.904136 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.904137 5300 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 00:43:19.904211 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.904143 5300 policy_none.go:44] "Start" Oct 30 00:43:19.911870 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.911830 5300 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 00:43:19.912916 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:43:19.912884 5300 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 00:43:19.914737 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.914719 5300 manager.go:341] "Starting Device Plugin manager" Oct 30 00:43:19.914868 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:43:19.914750 5300 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 00:43:19.914868 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.914774 5300 server.go:87] "Starting device plugin registration server" Oct 30 00:43:19.915096 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.915082 5300 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 00:43:19.915150 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.915100 5300 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 00:43:19.916918 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.916900 5300 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 00:43:19.917266 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.917226 5300 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 00:43:19.917354 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.917272 5300 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 00:43:19.917354 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.917280 5300 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 00:43:19.917422 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.917356 5300 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:43:19.917261814Z","UUID":"781af679-cc54-4aea-b004-f6b894db1a6b","Handler":null,"Name":"","Endpoint":""} Oct 30 00:43:19.918598 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.918578 5300 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 00:43:19.918692 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:19.918606 5300 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 00:43:19.919543 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:43:19.919522 5300 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 00:43:20.016170 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.016137 5300 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 00:43:20.018284 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.018262 5300 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-206.ec2.internal" event="NodeHasSufficientMemory" Oct 30 00:43:20.018403 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.018293 5300 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-206.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 00:43:20.018403 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.018307 5300 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-206.ec2.internal" event="NodeHasSufficientPID" Oct 30 00:43:20.018403 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.018351 5300 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-206.ec2.internal" Oct 30 00:43:20.054527 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.054490 5300 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-206.ec2.internal" Oct 30 00:43:20.054699 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.054570 5300 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-206.ec2.internal" Oct 30 00:43:20.113053 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.113005 5300 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-206.ec2.internal"] Oct 30 00:43:20.113539 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.113500 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-206.ec2.internal" event={"ID":"3b175f0644888f98ec9cdc35355eb1b8","Type":"ContainerStarted","Data":"f8616b8e19b9a4b4730f010b7c329b9922ac4e984ca6cbd319cc996f7bf50aac"} Oct 30 00:43:20.113648 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.113537 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-206.ec2.internal" event={"ID":"3b175f0644888f98ec9cdc35355eb1b8","Type":"ContainerStarted","Data":"8aedcc0431068aad1e5868f70d8904e299ca29410447c75871583ef000efb149"} Oct 30 00:43:20.113648 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.113608 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal" event={"ID":"91d0355fca0e17295f697c41968d06d7","Type":"ContainerStarted","Data":"b71f0b322f1e03a3d9d5b571df56131b5a19457428f9925e7b4093256229576e"} Oct 30 00:43:20.113648 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.113621 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal" event={"ID":"91d0355fca0e17295f697c41968d06d7","Type":"ContainerDied","Data":"ffbf7c1727a78c296c0d9d7895a47b4281c10c91811181dee30c24386b8d574c"} Oct 30 00:43:20.113648 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.113633 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal" event={"ID":"91d0355fca0e17295f697c41968d06d7","Type":"ContainerStarted","Data":"1a21ea02e80d4f6d786dbd8b2d11fb89a274e7c3ff0c3b317588934510b97e06"} Oct 30 00:43:20.113820 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.113729 5300 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal" Oct 30 00:43:20.113820 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.113733 5300 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-206.ec2.internal" Oct 30 00:43:20.141483 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.141415 5300 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:43:20.141483 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:43:20.141477 5300 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-206.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-206.ec2.internal" Oct 30 00:43:20.141840 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.141822 5300 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:43:20.141925 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:43:20.141863 5300 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal" Oct 30 00:43:20.223203 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.223175 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91d0355fca0e17295f697c41968d06d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal\" (UID: \"91d0355fca0e17295f697c41968d06d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal" Oct 30 00:43:20.223356 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.223209 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91d0355fca0e17295f697c41968d06d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal\" (UID: \"91d0355fca0e17295f697c41968d06d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal" Oct 30 00:43:20.223356 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.223237 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b175f0644888f98ec9cdc35355eb1b8-config\") pod \"kube-apiserver-proxy-ip-10-0-140-206.ec2.internal\" (UID: \"3b175f0644888f98ec9cdc35355eb1b8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-206.ec2.internal" Oct 30 00:43:20.323849 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.323811 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91d0355fca0e17295f697c41968d06d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal\" (UID: \"91d0355fca0e17295f697c41968d06d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal" Oct 30 00:43:20.323849 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.323849 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91d0355fca0e17295f697c41968d06d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal\" (UID: \"91d0355fca0e17295f697c41968d06d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal" Oct 30 00:43:20.324046 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.323865 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b175f0644888f98ec9cdc35355eb1b8-config\") pod \"kube-apiserver-proxy-ip-10-0-140-206.ec2.internal\" (UID: \"3b175f0644888f98ec9cdc35355eb1b8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-206.ec2.internal" Oct 30 00:43:20.324046 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.323913 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91d0355fca0e17295f697c41968d06d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal\" (UID: \"91d0355fca0e17295f697c41968d06d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal" Oct 30 00:43:20.324046 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.323921 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91d0355fca0e17295f697c41968d06d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal\" (UID: \"91d0355fca0e17295f697c41968d06d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal" Oct 30 00:43:20.324046 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.323982 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b175f0644888f98ec9cdc35355eb1b8-config\") pod \"kube-apiserver-proxy-ip-10-0-140-206.ec2.internal\" (UID: \"3b175f0644888f98ec9cdc35355eb1b8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-206.ec2.internal" Oct 30 00:43:20.756457 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.756426 5300 apiserver.go:52] "Watching apiserver" Oct 30 00:43:20.813301 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.813272 5300 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 00:43:20.813861 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.813839 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-xpngc","openshift-ingress-canary/ingress-canary-jpp4m","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal","kube-system/global-pull-secret-syncer-xdkdr","openshift-console/console-6cbd5c8c4-fbqjc","openshift-multus/multus-xmr6l","openshift-ovn-kubernetes/ovnkube-node-2nq94","kube-system/kube-apiserver-proxy-ip-10-0-140-206.ec2.internal","openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst","openshift-console/console-8698cc7897-jflvm","openshift-multus/network-metrics-daemon-przs7","openshift-dns/dns-default-5jhd7","openshift-monitoring/monitoring-plugin-77f4f76594-vmstj","openshift-monitoring/node-exporter-vngsd","kube-system/konnectivity-agent-89gxj","openshift-cluster-node-tuning-operator/tuned-pnxhn","openshift-insights/insights-runtime-extractor-xmsbl","openshift-monitoring/prometheus-k8s-0","openshift-multus/multus-additional-cni-plugins-5rhzn","openshift-network-diagnostics/network-check-target-c4b2t","openshift-dns/node-resolver-6mjml","openshift-image-registry/node-ca-g9ftg"] Oct 30 00:43:20.832008 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.831980 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 00:43:20.832129 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.832077 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tlfwv\"" Oct 30 00:43:20.832129 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.832101 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 00:43:20.832275 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.832249 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 00:43:20.832452 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.832436 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vdsv4\"" Oct 30 00:43:20.832583 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.832566 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7ppwp\"" Oct 30 00:43:20.832637 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.832570 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8tgxx\"" Oct 30 00:43:20.832688 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.832660 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hnsxr\"" Oct 30 00:43:20.832688 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.832678 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hp8gq\"" Oct 30 00:43:20.832806 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.832711 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 00:43:20.832806 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.832746 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-stkll\"" Oct 30 00:43:20.832806 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.832778 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 00:43:20.832935 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.832827 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 00:43:20.832935 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.832852 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 00:43:20.832935 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.832865 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:43:20.833051 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.832976 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 00:43:20.833051 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.832994 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 00:43:20.833051 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.832993 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 00:43:20.833051 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.833007 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 00:43:20.833051 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.833017 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 00:43:20.833051 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.833017 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 00:43:20.833051 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.833029 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 00:43:20.833285 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.833040 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 00:43:20.833285 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.833137 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 00:43:20.833285 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.833143 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 00:43:20.833285 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.833181 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 00:43:20.833285 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.833216 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 00:43:20.845124 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.845105 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 00:43:20.874421 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.874392 5300 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-206.ec2.internal" Oct 30 00:43:20.874568 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.874544 5300 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal" Oct 30 00:43:20.883667 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.883648 5300 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 00:43:20.902012 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.901991 5300 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:43:20.902012 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.902007 5300 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 00:43:20.902186 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:43:20.902037 5300 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-206.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-206.ec2.internal" Oct 30 00:43:20.902186 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:43:20.902072 5300 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-206.ec2.internal" Oct 30 00:43:20.919099 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.919074 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cbd5c8c4-fbqjc"] Oct 30 00:43:20.919314 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:43:20.919292 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[console-config console-oauth-config console-serving-cert kube-api-access-8mplm oauth-serving-cert service-ca tmp], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-console/console-6cbd5c8c4-fbqjc" podUID="ffa2afe9-832d-452f-855d-982f9e47a601" Oct 30 00:43:20.925943 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.925920 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcf5056b-ef4e-430b-8b5c-83fad65af440-metrics-client-ca\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:20.926039 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.926020 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e991db07-d748-4ced-8d7a-4266194e89d8-config-out\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.926097 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.926053 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rxlb\" (UniqueName: \"kubernetes.io/projected/da571014-afa3-4a40-a083-dc708bf83ba3-kube-api-access-9rxlb\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:20.926097 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.926072 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b22701a-4358-4fd4-92d9-bd838f5af629-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmsbl\" (UID: \"0b22701a-4358-4fd4-92d9-bd838f5af629\") " pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:43:20.926156 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.926092 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb9nw\" (UniqueName: \"kubernetes.io/projected/995884d0-cb39-4c07-9dbe-5cd3529d5908-kube-api-access-fb9nw\") pod \"node-resolver-6mjml\" (UID: \"995884d0-cb39-4c07-9dbe-5cd3529d5908\") " pod="openshift-dns/node-resolver-6mjml" Oct 30 00:43:20.926156 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.926119 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-service-ca\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:20.926156 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.926144 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlnvv\" (UniqueName: \"kubernetes.io/projected/49f04f3b-49d5-4c22-88a2-e33e106853ce-kube-api-access-tlnvv\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:43:20.926266 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.926155 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e991db07-d748-4ced-8d7a-4266194e89d8-config-out\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.926266 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.926169 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.926266 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.926207 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.926266 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.926255 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-var-lib-cni-multus\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:20.926422 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.926333 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-etc-openvswitch\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:20.926422 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.926381 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5dff88d3-2e35-40e9-8aea-208519485adc-host\") pod \"node-ca-g9ftg\" (UID: \"5dff88d3-2e35-40e9-8aea-208519485adc\") " pod="openshift-image-registry/node-ca-g9ftg" Oct 30 00:43:20.926486 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.926403 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-tls\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:20.926486 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.926441 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-kubelet\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:20.926486 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.926464 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/994c018e-e6c5-4d21-adac-9540676a67c0-ovnkube-script-lib\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:20.926617 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.926494 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-serving-cert\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:20.926617 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.926511 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2726g\" (UniqueName: \"kubernetes.io/projected/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-kube-api-access-2726g\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:20.926712 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.926628 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.926788 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.926712 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-sys\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:20.926788 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.926778 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-run-multus-certs\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:20.926859 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.926810 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-service-ca\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:20.926859 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.926834 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-sysctl-d\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:20.926958 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.926876 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/994c018e-e6c5-4d21-adac-9540676a67c0-ovnkube-script-lib\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:20.926958 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.926881 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.927063 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.927040 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-run-netns\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:20.927237 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.927090 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-run-openvswitch\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:20.927300 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.927240 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-config\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:20.927358 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.927317 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l7h2\" (UniqueName: \"kubernetes.io/projected/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-kube-api-access-4l7h2\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:43:20.927358 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.927345 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-systemd\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:20.927447 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.927374 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndpj2\" (UniqueName: \"kubernetes.io/projected/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-kube-api-access-ndpj2\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:20.927447 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.927415 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4gkr\" (UniqueName: \"kubernetes.io/projected/b3f55f18-4143-4806-b9e4-989e153f7e38-kube-api-access-m4gkr\") pod \"network-check-target-c4b2t\" (UID: \"b3f55f18-4143-4806-b9e4-989e153f7e38\") " pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:43:20.927538 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.927453 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49f04f3b-49d5-4c22-88a2-e33e106853ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:43:20.927538 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.927480 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8kj9\" (UniqueName: \"kubernetes.io/projected/653fb098-e2a1-4674-9baa-6975e9dd0fec-kube-api-access-s8kj9\") pod \"ingress-canary-jpp4m\" (UID: \"653fb098-e2a1-4674-9baa-6975e9dd0fec\") " pod="openshift-ingress-canary/ingress-canary-jpp4m" Oct 30 00:43:20.927682 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.927660 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81258c01-d653-4a05-87ca-8373e7fad83d-metrics-tls\") pod \"dns-default-5jhd7\" (UID: \"81258c01-d653-4a05-87ca-8373e7fad83d\") " pod="openshift-dns/dns-default-5jhd7" Oct 30 00:43:20.927740 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.927712 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcf5056b-ef4e-430b-8b5c-83fad65af440-sys\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:20.927809 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.927747 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.927809 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.927788 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.927905 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.927829 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-metrics-certs\") pod \"network-metrics-daemon-przs7\" (UID: \"1c2f1e93-f19e-4c0f-944e-7f52261f6e81\") " pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:43:20.927905 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.927851 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49f04f3b-49d5-4c22-88a2-e33e106853ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:43:20.927905 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.927873 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kghwp\" (UniqueName: \"kubernetes.io/projected/fcf5056b-ef4e-430b-8b5c-83fad65af440-kube-api-access-kghwp\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:20.928048 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.927911 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e991db07-d748-4ced-8d7a-4266194e89d8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.928048 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.927941 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:43:20.928048 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.927966 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-modprobe-d\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:20.928048 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.927984 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/653fb098-e2a1-4674-9baa-6975e9dd0fec-cert\") pod \"ingress-canary-jpp4m\" (UID: \"653fb098-e2a1-4674-9baa-6975e9dd0fec\") " pod="openshift-ingress-canary/ingress-canary-jpp4m" Oct 30 00:43:20.928048 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928033 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/67bde315-b2ea-4d17-88d8-f290cd2c994e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vmstj\" (UID: \"67bde315-b2ea-4d17-88d8-f290cd2c994e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmstj" Oct 30 00:43:20.928292 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928054 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/994c018e-e6c5-4d21-adac-9540676a67c0-ovn-node-metrics-cert\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:20.928292 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928070 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp68j\" (UniqueName: \"kubernetes.io/projected/67bde315-b2ea-4d17-88d8-f290cd2c994e-kube-api-access-mp68j\") pod \"monitoring-plugin-77f4f76594-vmstj\" (UID: \"67bde315-b2ea-4d17-88d8-f290cd2c994e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmstj" Oct 30 00:43:20.928292 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928089 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-web-config\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.928292 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928114 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-node-log\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:20.928292 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928138 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-trusted-ca-bundle\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:20.928292 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928162 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-sysctl-conf\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:20.928292 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928250 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/271e26db-6a8b-4e46-afb7-139cbc57c1e1-konnectivity-ca\") pod \"konnectivity-agent-89gxj\" (UID: \"271e26db-6a8b-4e46-afb7-139cbc57c1e1\") " pod="kube-system/konnectivity-agent-89gxj" Oct 30 00:43:20.928574 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928303 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-sys-fs\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:43:20.928574 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928329 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7127ca6b-3962-47a9-9089-07bcba277bc7-host-slash\") pod \"iptables-alerter-xpngc\" (UID: \"7127ca6b-3962-47a9-9089-07bcba277bc7\") " pod="openshift-network-operator/iptables-alerter-xpngc" Oct 30 00:43:20.928574 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928350 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/994c018e-e6c5-4d21-adac-9540676a67c0-ovn-node-metrics-cert\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:20.928574 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928373 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-multus-cni-dir\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:20.928574 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928402 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-run-systemd\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:20.928574 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928434 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/271e26db-6a8b-4e46-afb7-139cbc57c1e1-konnectivity-ca\") pod \"konnectivity-agent-89gxj\" (UID: \"271e26db-6a8b-4e46-afb7-139cbc57c1e1\") " pod="kube-system/konnectivity-agent-89gxj" Oct 30 00:43:20.928574 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928449 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9srn\" (UniqueName: \"kubernetes.io/projected/0b22701a-4358-4fd4-92d9-bd838f5af629-kube-api-access-b9srn\") pod \"insights-runtime-extractor-xmsbl\" (UID: \"0b22701a-4358-4fd4-92d9-bd838f5af629\") " pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:43:20.928574 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928514 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/81258c01-d653-4a05-87ca-8373e7fad83d-config-volume\") pod \"dns-default-5jhd7\" (UID: \"81258c01-d653-4a05-87ca-8373e7fad83d\") " pod="openshift-dns/dns-default-5jhd7" Oct 30 00:43:20.928574 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928547 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-var-lib-cni-bin\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:20.928574 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928572 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-multus-conf-dir\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:20.929001 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928611 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-tuned\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:20.929001 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928640 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-run-k8s-cni-cncf-io\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:20.929001 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928666 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-dbus\") pod \"global-pull-secret-syncer-xdkdr\" (UID: \"b0661ca3-6f0a-4744-9d50-f2baa1a80b45\") " pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:43:20.929001 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928691 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x2hb\" (UniqueName: \"kubernetes.io/projected/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-kube-api-access-9x2hb\") pod \"network-metrics-daemon-przs7\" (UID: \"1c2f1e93-f19e-4c0f-944e-7f52261f6e81\") " pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:43:20.929001 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928730 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-tuned\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:20.929001 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928740 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c78afce-0bbb-49e1-aceb-ae4a69844075-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-g9xp7\" (UID: \"3c78afce-0bbb-49e1-aceb-ae4a69844075\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7" Oct 30 00:43:20.929001 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928815 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwqqr\" (UniqueName: \"kubernetes.io/projected/3c78afce-0bbb-49e1-aceb-ae4a69844075-kube-api-access-bwqqr\") pod \"prometheus-operator-9cd89b74c-g9xp7\" (UID: \"3c78afce-0bbb-49e1-aceb-ae4a69844075\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7" Oct 30 00:43:20.929001 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928839 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ffa2afe9-832d-452f-855d-982f9e47a601-console-oauth-config\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:20.929001 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928859 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.929001 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928947 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.929001 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.928982 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-slash\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:20.929303 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.929020 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-lib-modules\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:20.929303 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.929063 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-config\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.929303 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.929163 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-multus-socket-dir-parent\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:20.929303 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.929192 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-hostroot\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:20.929303 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.929215 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-systemd-units\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:20.929303 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.929265 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49f04f3b-49d5-4c22-88a2-e33e106853ce-system-cni-dir\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:43:20.929303 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.929298 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-oauth-serving-cert\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:20.929580 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.929347 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj6jk\" (UniqueName: \"kubernetes.io/projected/7127ca6b-3962-47a9-9089-07bcba277bc7-kube-api-access-hj6jk\") pod \"iptables-alerter-xpngc\" (UID: \"7127ca6b-3962-47a9-9089-07bcba277bc7\") " pod="openshift-network-operator/iptables-alerter-xpngc" Oct 30 00:43:20.929580 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.929396 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da571014-afa3-4a40-a083-dc708bf83ba3-cni-binary-copy\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:20.929688 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.929616 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/994c018e-e6c5-4d21-adac-9540676a67c0-ovnkube-config\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:20.929688 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.929679 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da571014-afa3-4a40-a083-dc708bf83ba3-cni-binary-copy\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:20.929830 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.929723 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/994c018e-e6c5-4d21-adac-9540676a67c0-env-overrides\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:20.929899 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.929854 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/994c018e-e6c5-4d21-adac-9540676a67c0-ovnkube-config\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:20.929955 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.929897 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r7kr\" (UniqueName: \"kubernetes.io/projected/994c018e-e6c5-4d21-adac-9540676a67c0-kube-api-access-9r7kr\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:20.929955 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.929931 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/994c018e-e6c5-4d21-adac-9540676a67c0-env-overrides\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:20.930052 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.929976 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-kubelet-config\") pod \"global-pull-secret-syncer-xdkdr\" (UID: \"b0661ca3-6f0a-4744-9d50-f2baa1a80b45\") " pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:43:20.930052 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930015 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49f04f3b-49d5-4c22-88a2-e33e106853ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:43:20.930144 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930064 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49f04f3b-49d5-4c22-88a2-e33e106853ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:43:20.930144 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930110 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81258c01-d653-4a05-87ca-8373e7fad83d-tmp-dir\") pod \"dns-default-5jhd7\" (UID: \"81258c01-d653-4a05-87ca-8373e7fad83d\") " pod="openshift-dns/dns-default-5jhd7" Oct 30 00:43:20.930243 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930149 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.930243 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930181 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mghhs\" (UniqueName: \"kubernetes.io/projected/e991db07-d748-4ced-8d7a-4266194e89d8-kube-api-access-mghhs\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.930352 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930207 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-etc-selinux\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:43:20.930352 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930301 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-system-cni-dir\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:20.930352 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930221 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81258c01-d653-4a05-87ca-8373e7fad83d-tmp-dir\") pod \"dns-default-5jhd7\" (UID: \"81258c01-d653-4a05-87ca-8373e7fad83d\") " pod="openshift-dns/dns-default-5jhd7" Oct 30 00:43:20.930557 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930356 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.930557 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930399 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.930557 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930425 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/da571014-afa3-4a40-a083-dc708bf83ba3-multus-daemon-config\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:20.930557 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930458 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-oauth-config\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:20.930557 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930476 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-oauth-serving-cert\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:20.930557 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930482 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.930557 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930493 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-sysconfig\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:20.930557 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930534 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-cnibin\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:20.930979 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930560 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-var-lib-kubelet\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:20.930979 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930616 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/995884d0-cb39-4c07-9dbe-5cd3529d5908-hosts-file\") pod \"node-resolver-6mjml\" (UID: \"995884d0-cb39-4c07-9dbe-5cd3529d5908\") " pod="openshift-dns/node-resolver-6mjml" Oct 30 00:43:20.930979 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930631 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/da571014-afa3-4a40-a083-dc708bf83ba3-multus-daemon-config\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:20.930979 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930651 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5dff88d3-2e35-40e9-8aea-208519485adc-serviceca\") pod \"node-ca-g9ftg\" (UID: \"5dff88d3-2e35-40e9-8aea-208519485adc\") " pod="openshift-image-registry/node-ca-g9ftg" Oct 30 00:43:20.930979 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930691 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b74st\" (UniqueName: \"kubernetes.io/projected/81258c01-d653-4a05-87ca-8373e7fad83d-kube-api-access-b74st\") pod \"dns-default-5jhd7\" (UID: \"81258c01-d653-4a05-87ca-8373e7fad83d\") " pod="openshift-dns/dns-default-5jhd7" Oct 30 00:43:20.930979 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930723 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.930979 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930751 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.930979 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930800 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-cni-bin\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:20.930979 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930826 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/995884d0-cb39-4c07-9dbe-5cd3529d5908-tmp-dir\") pod \"node-resolver-6mjml\" (UID: \"995884d0-cb39-4c07-9dbe-5cd3529d5908\") " pod="openshift-dns/node-resolver-6mjml" Oct 30 00:43:20.930979 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930852 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w48mv\" (UniqueName: \"kubernetes.io/projected/5dff88d3-2e35-40e9-8aea-208519485adc-kube-api-access-w48mv\") pod \"node-ca-g9ftg\" (UID: \"5dff88d3-2e35-40e9-8aea-208519485adc\") " pod="openshift-image-registry/node-ca-g9ftg" Oct 30 00:43:20.930979 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930875 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-wtmp\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:20.930979 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930901 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-accelerators-collector-config\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:20.930979 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930930 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-var-lib-openvswitch\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:20.930979 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930955 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b22701a-4358-4fd4-92d9-bd838f5af629-crio-socket\") pod \"insights-runtime-extractor-xmsbl\" (UID: \"0b22701a-4358-4fd4-92d9-bd838f5af629\") " pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:43:20.930979 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.930981 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b22701a-4358-4fd4-92d9-bd838f5af629-data-volume\") pod \"insights-runtime-extractor-xmsbl\" (UID: \"0b22701a-4358-4fd4-92d9-bd838f5af629\") " pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:43:20.932030 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.931030 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-kubernetes\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:20.932030 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.931094 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-run\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:20.932030 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.931129 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-host\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:20.932030 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.931135 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/995884d0-cb39-4c07-9dbe-5cd3529d5908-tmp-dir\") pod \"node-resolver-6mjml\" (UID: \"995884d0-cb39-4c07-9dbe-5cd3529d5908\") " pod="openshift-dns/node-resolver-6mjml" Oct 30 00:43:20.932030 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.931203 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b22701a-4358-4fd4-92d9-bd838f5af629-data-volume\") pod \"insights-runtime-extractor-xmsbl\" (UID: \"0b22701a-4358-4fd4-92d9-bd838f5af629\") " pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:43:20.932030 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.931410 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 00:43:20.932030 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.931447 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 00:43:20.932030 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.931473 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5dff88d3-2e35-40e9-8aea-208519485adc-serviceca\") pod \"node-ca-g9ftg\" (UID: \"5dff88d3-2e35-40e9-8aea-208519485adc\") " pod="openshift-image-registry/node-ca-g9ftg" Oct 30 00:43:20.932030 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.931534 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/271e26db-6a8b-4e46-afb7-139cbc57c1e1-agent-certs\") pod \"konnectivity-agent-89gxj\" (UID: \"271e26db-6a8b-4e46-afb7-139cbc57c1e1\") " pod="kube-system/konnectivity-agent-89gxj" Oct 30 00:43:20.932030 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.931581 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-cni-netd\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:20.932030 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.931644 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 00:43:20.932030 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.931659 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:20.932030 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.931699 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.932030 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.931707 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 00:43:20.932030 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.931746 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 00:43:20.932030 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.931780 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.932030 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.931819 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-original-pull-secret\") pod \"global-pull-secret-syncer-xdkdr\" (UID: \"b0661ca3-6f0a-4744-9d50-f2baa1a80b45\") " pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:43:20.932030 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.931746 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 00:43:20.932030 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.931851 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mplm\" (UniqueName: \"kubernetes.io/projected/ffa2afe9-832d-452f-855d-982f9e47a601-kube-api-access-8mplm\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:20.932030 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.931849 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 00:43:20.932030 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.931922 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kfrd4\"" Oct 30 00:43:20.933050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.932081 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rbt7f\"" Oct 30 00:43:20.933050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.932095 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 00:43:20.933050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.932114 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:20.933050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.932152 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b22701a-4358-4fd4-92d9-bd838f5af629-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmsbl\" (UID: \"0b22701a-4358-4fd4-92d9-bd838f5af629\") " pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:43:20.933050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.932181 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 00:43:20.933050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.932191 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffa2afe9-832d-452f-855d-982f9e47a601-tmp\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:20.933050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.932224 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-run-netns\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:20.933050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.932341 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 00:43:20.933050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.932367 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-run-ovn\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:20.933050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.932389 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffa2afe9-832d-452f-855d-982f9e47a601-tmp\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:20.933050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.932427 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:20.933050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.932457 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-console-config\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:20.933050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.932520 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 00:43:20.933050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.932608 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fcf5056b-ef4e-430b-8b5c-83fad65af440-root\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:20.933050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.932698 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7127ca6b-3962-47a9-9089-07bcba277bc7-iptables-alerter-script\") pod \"iptables-alerter-xpngc\" (UID: \"7127ca6b-3962-47a9-9089-07bcba277bc7\") " pod="openshift-network-operator/iptables-alerter-xpngc" Oct 30 00:43:20.933050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.932734 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 00:43:20.933050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.932753 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-log-socket\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:20.933050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.932806 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c78afce-0bbb-49e1-aceb-ae4a69844075-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-g9xp7\" (UID: \"3c78afce-0bbb-49e1-aceb-ae4a69844075\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7" Oct 30 00:43:20.933050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.932829 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-original-pull-secret\") pod \"global-pull-secret-syncer-xdkdr\" (UID: \"b0661ca3-6f0a-4744-9d50-f2baa1a80b45\") " pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:43:20.933050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.932839 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kcfzd\"" Oct 30 00:43:20.933050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.932865 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-tmp\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:20.933050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.932895 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-p4p9c\"" Oct 30 00:43:20.933050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.932933 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49f04f3b-49d5-4c22-88a2-e33e106853ce-cnibin\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:43:20.933050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.932994 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-tmp\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:20.933050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933000 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-789mk\"" Oct 30 00:43:20.933050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933047 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933065 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49f04f3b-49d5-4c22-88a2-e33e106853ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933105 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-socket-dir\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933072 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v7mtf\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933177 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933187 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-device-dir\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933207 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vhshc\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933219 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-var-lib-kubelet\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933252 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933180 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933250 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-etc-kubernetes\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933309 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c78afce-0bbb-49e1-aceb-ae4a69844075-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-g9xp7\" (UID: \"3c78afce-0bbb-49e1-aceb-ae4a69844075\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933252 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933460 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933478 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ffa2afe9-832d-452f-855d-982f9e47a601-console-serving-cert\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933497 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933498 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933526 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-textfile\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933555 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-registration-dir\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933591 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-hz7cw\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933594 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933615 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-run-ovn-kubernetes\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933645 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-tmp\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933661 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933725 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933739 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933780 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933823 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933881 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933902 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933730 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933956 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-textfile\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933964 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ffa2afe9-832d-452f-855d-982f9e47a601-console-serving-cert\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.934014 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.933621 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.934028 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/271e26db-6a8b-4e46-afb7-139cbc57c1e1-agent-certs\") pod \"konnectivity-agent-89gxj\" (UID: \"271e26db-6a8b-4e46-afb7-139cbc57c1e1\") " pod="kube-system/konnectivity-agent-89gxj" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.934017 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.934045 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.934048 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.934075 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-tmp\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.934090 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.934100 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.934130 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.934143 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8uop28j2hbr7f\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.934148 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cd6zv\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.934196 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.934202 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.934218 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ph2zs\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.934221 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.934243 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.934254 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.934278 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5gz8r\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.934285 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:43:20.934263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.934297 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dzgqx\"" Oct 30 00:43:20.935679 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.934321 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 00:43:20.935679 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.934322 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b5pp6\"" Oct 30 00:43:20.936672 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.936653 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.936672 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.936665 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-service-ca\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:20.936820 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.936679 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b22701a-4358-4fd4-92d9-bd838f5af629-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmsbl\" (UID: \"0b22701a-4358-4fd4-92d9-bd838f5af629\") " pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:43:20.936820 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.936748 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-tls\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:20.936890 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.936857 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.936921 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.936911 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-serving-cert\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:20.937877 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.937859 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-service-ca\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:20.937985 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.937971 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-config\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:20.938020 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.937973 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81258c01-d653-4a05-87ca-8373e7fad83d-metrics-tls\") pod \"dns-default-5jhd7\" (UID: \"81258c01-d653-4a05-87ca-8373e7fad83d\") " pod="openshift-dns/dns-default-5jhd7" Oct 30 00:43:20.938114 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.938099 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e991db07-d748-4ced-8d7a-4266194e89d8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.938114 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.938107 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-metrics-certs\") pod \"network-metrics-daemon-przs7\" (UID: \"1c2f1e93-f19e-4c0f-944e-7f52261f6e81\") " pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:43:20.938245 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.938226 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.938299 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.938246 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/653fb098-e2a1-4674-9baa-6975e9dd0fec-cert\") pod \"ingress-canary-jpp4m\" (UID: \"653fb098-e2a1-4674-9baa-6975e9dd0fec\") " pod="openshift-ingress-canary/ingress-canary-jpp4m" Oct 30 00:43:20.938334 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.938314 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/67bde315-b2ea-4d17-88d8-f290cd2c994e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-vmstj\" (UID: \"67bde315-b2ea-4d17-88d8-f290cd2c994e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmstj" Oct 30 00:43:20.938384 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.938372 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-web-config\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.938695 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.938682 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.938845 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.938828 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/81258c01-d653-4a05-87ca-8373e7fad83d-config-volume\") pod \"dns-default-5jhd7\" (UID: \"81258c01-d653-4a05-87ca-8373e7fad83d\") " pod="openshift-dns/dns-default-5jhd7" Oct 30 00:43:20.938951 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.938935 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c78afce-0bbb-49e1-aceb-ae4a69844075-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-g9xp7\" (UID: \"3c78afce-0bbb-49e1-aceb-ae4a69844075\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7" Oct 30 00:43:20.939064 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.939047 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ffa2afe9-832d-452f-855d-982f9e47a601-console-oauth-config\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:20.939107 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.939053 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.939212 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.939200 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.939524 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.939511 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-config\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.939650 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.939636 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-oauth-serving-cert\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:20.941364 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.941272 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.941364 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.941274 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.941364 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.941309 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49f04f3b-49d5-4c22-88a2-e33e106853ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:43:20.941364 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.941320 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-oauth-config\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:20.941364 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.941310 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-oauth-serving-cert\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:20.941364 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.941325 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-accelerators-collector-config\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:20.942520 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.942502 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.942637 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.942611 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.942717 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.942655 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b22701a-4358-4fd4-92d9-bd838f5af629-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmsbl\" (UID: \"0b22701a-4358-4fd4-92d9-bd838f5af629\") " pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:43:20.942717 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.942683 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.943113 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.943086 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-console-config\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:20.943883 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.943863 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c78afce-0bbb-49e1-aceb-ae4a69844075-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-g9xp7\" (UID: \"3c78afce-0bbb-49e1-aceb-ae4a69844075\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7" Oct 30 00:43:20.943961 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.943949 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49f04f3b-49d5-4c22-88a2-e33e106853ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:43:20.944741 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.944722 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 00:43:20.944908 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.944808 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 00:43:20.944908 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.944841 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 00:43:20.948977 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.945576 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 00:43:20.948977 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.948929 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.948977 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.948936 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcf5056b-ef4e-430b-8b5c-83fad65af440-metrics-client-ca\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:20.950788 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.950328 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 00:43:20.950788 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.950731 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rxlb\" (UniqueName: \"kubernetes.io/projected/da571014-afa3-4a40-a083-dc708bf83ba3-kube-api-access-9rxlb\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:20.950930 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.950867 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlnvv\" (UniqueName: \"kubernetes.io/projected/49f04f3b-49d5-4c22-88a2-e33e106853ce-kube-api-access-tlnvv\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:43:20.950930 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.950900 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndpj2\" (UniqueName: \"kubernetes.io/projected/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-kube-api-access-ndpj2\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:20.951035 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.951012 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2726g\" (UniqueName: \"kubernetes.io/projected/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-kube-api-access-2726g\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:20.952021 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.951179 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 00:43:20.952021 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.951202 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp68j\" (UniqueName: \"kubernetes.io/projected/67bde315-b2ea-4d17-88d8-f290cd2c994e-kube-api-access-mp68j\") pod \"monitoring-plugin-77f4f76594-vmstj\" (UID: \"67bde315-b2ea-4d17-88d8-f290cd2c994e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmstj" Oct 30 00:43:20.952021 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.951236 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 00:43:20.952021 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.951412 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r7kr\" (UniqueName: \"kubernetes.io/projected/994c018e-e6c5-4d21-adac-9540676a67c0-kube-api-access-9r7kr\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:20.952021 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.951621 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9srn\" (UniqueName: \"kubernetes.io/projected/0b22701a-4358-4fd4-92d9-bd838f5af629-kube-api-access-b9srn\") pod \"insights-runtime-extractor-xmsbl\" (UID: \"0b22701a-4358-4fd4-92d9-bd838f5af629\") " pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:43:20.952021 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.951629 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l7h2\" (UniqueName: \"kubernetes.io/projected/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-kube-api-access-4l7h2\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:43:20.952021 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.951649 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4gkr\" (UniqueName: \"kubernetes.io/projected/b3f55f18-4143-4806-b9e4-989e153f7e38-kube-api-access-m4gkr\") pod \"network-check-target-c4b2t\" (UID: \"b3f55f18-4143-4806-b9e4-989e153f7e38\") " pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:43:20.952021 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.951708 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x2hb\" (UniqueName: \"kubernetes.io/projected/1c2f1e93-f19e-4c0f-944e-7f52261f6e81-kube-api-access-9x2hb\") pod \"network-metrics-daemon-przs7\" (UID: \"1c2f1e93-f19e-4c0f-944e-7f52261f6e81\") " pod="openshift-multus/network-metrics-daemon-przs7" Oct 30 00:43:20.952021 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.951973 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8kj9\" (UniqueName: \"kubernetes.io/projected/653fb098-e2a1-4674-9baa-6975e9dd0fec-kube-api-access-s8kj9\") pod \"ingress-canary-jpp4m\" (UID: \"653fb098-e2a1-4674-9baa-6975e9dd0fec\") " pod="openshift-ingress-canary/ingress-canary-jpp4m" Oct 30 00:43:20.952021 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.951993 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kghwp\" (UniqueName: \"kubernetes.io/projected/fcf5056b-ef4e-430b-8b5c-83fad65af440-kube-api-access-kghwp\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:20.952021 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.951994 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.952021 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.952022 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwqqr\" (UniqueName: \"kubernetes.io/projected/3c78afce-0bbb-49e1-aceb-ae4a69844075-kube-api-access-bwqqr\") pod \"prometheus-operator-9cd89b74c-g9xp7\" (UID: \"3c78afce-0bbb-49e1-aceb-ae4a69844075\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7" Oct 30 00:43:20.952381 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.952163 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj6jk\" (UniqueName: \"kubernetes.io/projected/7127ca6b-3962-47a9-9089-07bcba277bc7-kube-api-access-hj6jk\") pod \"iptables-alerter-xpngc\" (UID: \"7127ca6b-3962-47a9-9089-07bcba277bc7\") " pod="openshift-network-operator/iptables-alerter-xpngc" Oct 30 00:43:20.952381 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.952284 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mghhs\" (UniqueName: \"kubernetes.io/projected/e991db07-d748-4ced-8d7a-4266194e89d8-kube-api-access-mghhs\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:20.952480 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.952465 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb9nw\" (UniqueName: \"kubernetes.io/projected/995884d0-cb39-4c07-9dbe-5cd3529d5908-kube-api-access-fb9nw\") pod \"node-resolver-6mjml\" (UID: \"995884d0-cb39-4c07-9dbe-5cd3529d5908\") " pod="openshift-dns/node-resolver-6mjml" Oct 30 00:43:20.953219 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.953198 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c78afce-0bbb-49e1-aceb-ae4a69844075-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-g9xp7\" (UID: \"3c78afce-0bbb-49e1-aceb-ae4a69844075\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-g9xp7" Oct 30 00:43:20.953292 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.953267 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7127ca6b-3962-47a9-9089-07bcba277bc7-iptables-alerter-script\") pod \"iptables-alerter-xpngc\" (UID: \"7127ca6b-3962-47a9-9089-07bcba277bc7\") " pod="openshift-network-operator/iptables-alerter-xpngc" Oct 30 00:43:20.958519 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.958498 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-trusted-ca-bundle\") pod \"console-8698cc7897-jflvm\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:20.958800 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.958780 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b74st\" (UniqueName: \"kubernetes.io/projected/81258c01-d653-4a05-87ca-8373e7fad83d-kube-api-access-b74st\") pod \"dns-default-5jhd7\" (UID: \"81258c01-d653-4a05-87ca-8373e7fad83d\") " pod="openshift-dns/dns-default-5jhd7" Oct 30 00:43:20.958930 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.958909 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w48mv\" (UniqueName: \"kubernetes.io/projected/5dff88d3-2e35-40e9-8aea-208519485adc-kube-api-access-w48mv\") pod \"node-ca-g9ftg\" (UID: \"5dff88d3-2e35-40e9-8aea-208519485adc\") " pod="openshift-image-registry/node-ca-g9ftg" Oct 30 00:43:20.959102 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.959089 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mplm\" (UniqueName: \"kubernetes.io/projected/ffa2afe9-832d-452f-855d-982f9e47a601-kube-api-access-8mplm\") pod \"console-6cbd5c8c4-fbqjc\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:20.961081 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:20.961064 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:21.008970 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.008901 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:21.034046 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034024 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-cni-netd\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.034140 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034052 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.034140 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034069 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-run-netns\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.034140 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034085 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-run-ovn\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.034140 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034106 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fcf5056b-ef4e-430b-8b5c-83fad65af440-root\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:21.034140 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034121 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.034140 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034129 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-log-socket\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.034407 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034141 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-cni-netd\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.034407 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034155 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-run-netns\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.034407 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034165 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-run-ovn\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.034407 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034162 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49f04f3b-49d5-4c22-88a2-e33e106853ce-cnibin\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:43:21.034407 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034190 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-log-socket\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.034407 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034195 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49f04f3b-49d5-4c22-88a2-e33e106853ce-cnibin\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:43:21.034407 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034204 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-socket-dir\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:43:21.034407 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034190 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fcf5056b-ef4e-430b-8b5c-83fad65af440-root\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:21.034407 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034221 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-device-dir\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:43:21.034407 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034238 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-var-lib-kubelet\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:21.034407 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034253 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-etc-kubernetes\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:21.034407 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034256 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-socket-dir\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:43:21.034407 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034276 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-registration-dir\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:43:21.034407 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034292 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-run-ovn-kubernetes\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.034407 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034297 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-var-lib-kubelet\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:21.034407 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034314 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-device-dir\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:43:21.034407 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034327 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-etc-kubernetes\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:21.034407 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034313 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-var-lib-cni-multus\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:21.034407 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034356 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-var-lib-cni-multus\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:21.034407 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034358 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-run-ovn-kubernetes\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.034407 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034367 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-registration-dir\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:43:21.034407 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034362 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-etc-openvswitch\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.034407 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034394 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5dff88d3-2e35-40e9-8aea-208519485adc-host\") pod \"node-ca-g9ftg\" (UID: \"5dff88d3-2e35-40e9-8aea-208519485adc\") " pod="openshift-image-registry/node-ca-g9ftg" Oct 30 00:43:21.034407 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034395 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-etc-openvswitch\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.034407 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034416 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-kubelet\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034444 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-sys\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034454 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-kubelet\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034445 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5dff88d3-2e35-40e9-8aea-208519485adc-host\") pod \"node-ca-g9ftg\" (UID: \"5dff88d3-2e35-40e9-8aea-208519485adc\") " pod="openshift-image-registry/node-ca-g9ftg" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034468 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-run-multus-certs\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034483 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-sys\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034493 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-sysctl-d\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034514 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-run-netns\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034529 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-run-openvswitch\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034544 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-systemd\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034566 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcf5056b-ef4e-430b-8b5c-83fad65af440-sys\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034580 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-run-netns\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034590 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-run-openvswitch\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034609 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034617 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-run-multus-certs\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034627 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcf5056b-ef4e-430b-8b5c-83fad65af440-sys\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034634 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-sysctl-d\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034634 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-modprobe-d\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034609 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-systemd\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034655 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034666 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-node-log\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034685 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-sysctl-conf\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034705 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-sys-fs\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034723 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7127ca6b-3962-47a9-9089-07bcba277bc7-host-slash\") pod \"iptables-alerter-xpngc\" (UID: \"7127ca6b-3962-47a9-9089-07bcba277bc7\") " pod="openshift-network-operator/iptables-alerter-xpngc" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034730 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-node-log\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034743 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-multus-cni-dir\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034752 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-sys-fs\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034773 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-modprobe-d\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034787 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-run-systemd\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034814 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-var-lib-cni-bin\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034823 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7127ca6b-3962-47a9-9089-07bcba277bc7-host-slash\") pod \"iptables-alerter-xpngc\" (UID: \"7127ca6b-3962-47a9-9089-07bcba277bc7\") " pod="openshift-network-operator/iptables-alerter-xpngc" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034830 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-sysctl-conf\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034814 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-multus-cni-dir\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034834 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-run-systemd\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034830 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-multus-conf-dir\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034850 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-multus-conf-dir\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034867 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-var-lib-cni-bin\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034869 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-run-k8s-cni-cncf-io\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034892 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-dbus\") pod \"global-pull-secret-syncer-xdkdr\" (UID: \"b0661ca3-6f0a-4744-9d50-f2baa1a80b45\") " pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034909 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-slash\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034924 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-lib-modules\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034940 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-multus-socket-dir-parent\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034959 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-dbus\") pod \"global-pull-secret-syncer-xdkdr\" (UID: \"b0661ca3-6f0a-4744-9d50-f2baa1a80b45\") " pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034963 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-hostroot\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034960 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-slash\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034981 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-host-run-k8s-cni-cncf-io\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034996 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-systemd-units\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035008 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-multus-socket-dir-parent\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035016 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49f04f3b-49d5-4c22-88a2-e33e106853ce-system-cni-dir\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035019 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-systemd-units\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035022 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-lib-modules\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.034997 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-hostroot\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035038 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49f04f3b-49d5-4c22-88a2-e33e106853ce-system-cni-dir\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035055 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-kubelet-config\") pod \"global-pull-secret-syncer-xdkdr\" (UID: \"b0661ca3-6f0a-4744-9d50-f2baa1a80b45\") " pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035075 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49f04f3b-49d5-4c22-88a2-e33e106853ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035085 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0661ca3-6f0a-4744-9d50-f2baa1a80b45-kubelet-config\") pod \"global-pull-secret-syncer-xdkdr\" (UID: \"b0661ca3-6f0a-4744-9d50-f2baa1a80b45\") " pod="kube-system/global-pull-secret-syncer-xdkdr" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035093 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-etc-selinux\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035108 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-system-cni-dir\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035125 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-sysconfig\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035123 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49f04f3b-49d5-4c22-88a2-e33e106853ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5rhzn\" (UID: \"49f04f3b-49d5-4c22-88a2-e33e106853ce\") " pod="openshift-multus/multus-additional-cni-plugins-5rhzn" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035138 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1f1c6441-4ebb-48d6-9e09-bf69e431eb57-etc-selinux\") pod \"aws-ebs-csi-driver-node-rfmst\" (UID: \"1f1c6441-4ebb-48d6-9e09-bf69e431eb57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfmst" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035152 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-system-cni-dir\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035142 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-cnibin\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035165 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-sysconfig\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035176 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da571014-afa3-4a40-a083-dc708bf83ba3-cnibin\") pod \"multus-xmr6l\" (UID: \"da571014-afa3-4a40-a083-dc708bf83ba3\") " pod="openshift-multus/multus-xmr6l" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035190 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-var-lib-kubelet\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035207 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-var-lib-kubelet\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035214 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/995884d0-cb39-4c07-9dbe-5cd3529d5908-hosts-file\") pod \"node-resolver-6mjml\" (UID: \"995884d0-cb39-4c07-9dbe-5cd3529d5908\") " pod="openshift-dns/node-resolver-6mjml" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035243 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-cni-bin\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035265 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/995884d0-cb39-4c07-9dbe-5cd3529d5908-hosts-file\") pod \"node-resolver-6mjml\" (UID: \"995884d0-cb39-4c07-9dbe-5cd3529d5908\") " pod="openshift-dns/node-resolver-6mjml" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035269 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-wtmp\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035288 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-host-cni-bin\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035293 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-var-lib-openvswitch\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035316 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fcf5056b-ef4e-430b-8b5c-83fad65af440-node-exporter-wtmp\") pod \"node-exporter-vngsd\" (UID: \"fcf5056b-ef4e-430b-8b5c-83fad65af440\") " pod="openshift-monitoring/node-exporter-vngsd" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035316 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b22701a-4358-4fd4-92d9-bd838f5af629-crio-socket\") pod \"insights-runtime-extractor-xmsbl\" (UID: \"0b22701a-4358-4fd4-92d9-bd838f5af629\") " pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035341 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-kubernetes\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035350 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/994c018e-e6c5-4d21-adac-9540676a67c0-var-lib-openvswitch\") pod \"ovnkube-node-2nq94\" (UID: \"994c018e-e6c5-4d21-adac-9540676a67c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035356 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-run\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035379 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-run\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035380 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-host\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:21.035362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035406 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-etc-kubernetes\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:21.037512 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035433 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b22701a-4358-4fd4-92d9-bd838f5af629-crio-socket\") pod \"insights-runtime-extractor-xmsbl\" (UID: \"0b22701a-4358-4fd4-92d9-bd838f5af629\") " pod="openshift-insights/insights-runtime-extractor-xmsbl" Oct 30 00:43:21.037512 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.035445 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0-host\") pod \"tuned-pnxhn\" (UID: \"f9d9dbc1-6e85-441e-8eaa-88c5e95b75a0\") " pod="openshift-cluster-node-tuning-operator/tuned-pnxhn" Oct 30 00:43:21.092154 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.092117 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:21.123891 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.123870 5300 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:43:21.602726 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:43:21.602687 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:43:21.880617 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.880515 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e991db07-d748-4ced-8d7a-4266194e89d8","Type":"ContainerStarted","Data":"eb7240966081b8adabf0816e2a664cd069bd5e95134fc0b9e7cf0c3b37bfe767"} Oct 30 00:43:21.880617 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.880597 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e991db07-d748-4ced-8d7a-4266194e89d8","Type":"ContainerStarted","Data":"6641e039709bb8422093c46dee9a083e9985c4918367fb0856c71dbd1215b86e"} Oct 30 00:43:21.934657 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:21.934627 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:22.875199 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:22.875168 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5jhd7" Oct 30 00:43:23.864256 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:23.864228 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:23.867888 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:23.867867 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:23.890582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:23.890548 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e991db07-d748-4ced-8d7a-4266194e89d8","Type":"ContainerStarted","Data":"22e127c9764069b72c51267dcfbe5fb0a4aca984d9a03661f64f42c198ec4571"} Oct 30 00:43:23.890720 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:23.890628 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e991db07-d748-4ced-8d7a-4266194e89d8","Type":"ContainerStarted","Data":"ef4b00c799ec95588f03c54a29a6477e47df634086638ec09304928e4ccf470f"} Oct 30 00:43:23.890720 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:23.890668 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e991db07-d748-4ced-8d7a-4266194e89d8","Type":"ContainerStarted","Data":"9462859e84a98d3bdbe6ffd54831d6576829ba911f5f0c69fc66527fab52787f"} Oct 30 00:43:23.890720 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:23.890702 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e991db07-d748-4ced-8d7a-4266194e89d8","Type":"ContainerStarted","Data":"2d505ff75a3bcd1c649feb088b62ddd264d0c46069cfb4b7de39b9c367692c1c"} Oct 30 00:43:25.695824 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:25.695787 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmstj" Oct 30 00:43:25.700007 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:25.699984 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-vmstj" Oct 30 00:43:26.134118 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:26.134080 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:26.255101 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:26.255070 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:26.458061 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:26.457971 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:26.662557 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:26.662517 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:26.896481 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:26.896446 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:27.880000 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:27.879957 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:43:28.133659 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:28.133580 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:28.240726 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:28.240691 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:29.334118 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:29.334064 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:29.434598 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:29.434564 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:30.215231 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:30.215205 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:30.215483 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:30.215466 5300 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 00:43:30.767665 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:30.767580 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:30.886363 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:30.886311 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:34.136750 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:34.136714 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2nq94" Oct 30 00:43:36.667919 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:36.667886 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:43:37.441409 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:37.441376 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:39.188296 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:39.188267 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:39.289974 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:39.289931 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:39.390561 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:39.390524 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:39.490338 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:39.490306 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:45.713672 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:45.713638 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:45.794418 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:45.794385 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:46.901670 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:46.901624 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6cbd5c8c4-fbqjc" podUID="ffa2afe9-832d-452f-855d-982f9e47a601" containerName="console" containerID="cri-o://428cc127c34e536430a5519d7f3bb47121127e337a6706d13c5722e96ae9062b" gracePeriod=15 Oct 30 00:43:47.180318 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.180246 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:47.255891 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.255850 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-service-ca\") pod \"ffa2afe9-832d-452f-855d-982f9e47a601\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " Oct 30 00:43:47.256060 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.255904 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffa2afe9-832d-452f-855d-982f9e47a601-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ffa2afe9-832d-452f-855d-982f9e47a601-console-serving-cert\") pod \"ffa2afe9-832d-452f-855d-982f9e47a601\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " Oct 30 00:43:47.256060 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.255982 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-oauth-serving-cert\") pod \"ffa2afe9-832d-452f-855d-982f9e47a601\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " Oct 30 00:43:47.256060 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.256020 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-console-config\" (UniqueName: \"kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-console-config\") pod \"ffa2afe9-832d-452f-855d-982f9e47a601\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " Oct 30 00:43:47.256230 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.256059 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffa2afe9-832d-452f-855d-982f9e47a601-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ffa2afe9-832d-452f-855d-982f9e47a601-console-oauth-config\") pod \"ffa2afe9-832d-452f-855d-982f9e47a601\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " Oct 30 00:43:47.256230 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.256110 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffa2afe9-832d-452f-855d-982f9e47a601-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffa2afe9-832d-452f-855d-982f9e47a601-tmp\") pod \"ffa2afe9-832d-452f-855d-982f9e47a601\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " Oct 30 00:43:47.256344 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.256266 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffa2afe9-832d-452f-855d-982f9e47a601-kube-api-access-8mplm\" (UniqueName: \"kubernetes.io/projected/ffa2afe9-832d-452f-855d-982f9e47a601-kube-api-access-8mplm\") pod \"ffa2afe9-832d-452f-855d-982f9e47a601\" (UID: \"ffa2afe9-832d-452f-855d-982f9e47a601\") " Oct 30 00:43:47.256400 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.256371 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-service-ca" pod "ffa2afe9-832d-452f-855d-982f9e47a601" (UID: "ffa2afe9-832d-452f-855d-982f9e47a601"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:43:47.256451 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.256391 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-oauth-serving-cert" pod "ffa2afe9-832d-452f-855d-982f9e47a601" (UID: "ffa2afe9-832d-452f-855d-982f9e47a601"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:43:47.256551 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.256532 5300 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:43:47.256641 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.256556 5300 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-oauth-serving-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:43:47.256641 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.256577 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa2afe9-832d-452f-855d-982f9e47a601-tmp" pod "ffa2afe9-832d-452f-855d-982f9e47a601" (UID: "ffa2afe9-832d-452f-855d-982f9e47a601"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:43:47.256705 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.256644 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-console-config" pod "ffa2afe9-832d-452f-855d-982f9e47a601" (UID: "ffa2afe9-832d-452f-855d-982f9e47a601"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:43:47.258364 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.258335 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffa2afe9-832d-452f-855d-982f9e47a601-console-serving-cert" pod "ffa2afe9-832d-452f-855d-982f9e47a601" (UID: "ffa2afe9-832d-452f-855d-982f9e47a601"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:43:47.258527 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.258503 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffa2afe9-832d-452f-855d-982f9e47a601-kube-api-access-8mplm" pod "ffa2afe9-832d-452f-855d-982f9e47a601" (UID: "ffa2afe9-832d-452f-855d-982f9e47a601"). InnerVolumeSpecName "kube-api-access-8mplm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:43:47.258588 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.258521 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffa2afe9-832d-452f-855d-982f9e47a601-console-oauth-config" pod "ffa2afe9-832d-452f-855d-982f9e47a601" (UID: "ffa2afe9-832d-452f-855d-982f9e47a601"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:43:47.357164 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.357122 5300 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ffa2afe9-832d-452f-855d-982f9e47a601-console-config\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:43:47.357164 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.357165 5300 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ffa2afe9-832d-452f-855d-982f9e47a601-console-oauth-config\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:43:47.357379 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.357182 5300 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffa2afe9-832d-452f-855d-982f9e47a601-tmp\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:43:47.357379 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.357197 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mplm\" (UniqueName: \"kubernetes.io/projected/ffa2afe9-832d-452f-855d-982f9e47a601-kube-api-access-8mplm\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:43:47.357379 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.357213 5300 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ffa2afe9-832d-452f-855d-982f9e47a601-console-serving-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:43:47.870288 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.869008 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:47.882470 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.882446 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-c4b2t" Oct 30 00:43:47.923870 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.923845 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:47.969775 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.969735 5300 generic.go:356] "Generic (PLEG): container finished" podID="ffa2afe9-832d-452f-855d-982f9e47a601" containerID="428cc127c34e536430a5519d7f3bb47121127e337a6706d13c5722e96ae9062b" exitCode=0 Oct 30 00:43:47.969946 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.969799 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cbd5c8c4-fbqjc" Oct 30 00:43:47.969946 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.969794 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cbd5c8c4-fbqjc" event={"ID":"ffa2afe9-832d-452f-855d-982f9e47a601","Type":"ContainerDied","Data":"428cc127c34e536430a5519d7f3bb47121127e337a6706d13c5722e96ae9062b"} Oct 30 00:43:47.970064 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.969995 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cbd5c8c4-fbqjc" event={"ID":"ffa2afe9-832d-452f-855d-982f9e47a601","Type":"ContainerDied","Data":"9b3abef169ea86e60b9acf2249c3cb82e1023cf6d9e242acaa4994c5af54e583"} Oct 30 00:43:47.970064 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.970032 5300 scope.go:122] "RemoveContainer" containerID="428cc127c34e536430a5519d7f3bb47121127e337a6706d13c5722e96ae9062b" Oct 30 00:43:47.978572 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.978549 5300 scope.go:122] "RemoveContainer" containerID="428cc127c34e536430a5519d7f3bb47121127e337a6706d13c5722e96ae9062b" Oct 30 00:43:47.978874 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:43:47.978839 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"428cc127c34e536430a5519d7f3bb47121127e337a6706d13c5722e96ae9062b\": container with ID starting with 428cc127c34e536430a5519d7f3bb47121127e337a6706d13c5722e96ae9062b not found: ID does not exist" containerID="428cc127c34e536430a5519d7f3bb47121127e337a6706d13c5722e96ae9062b" Oct 30 00:43:47.978965 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.978882 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"428cc127c34e536430a5519d7f3bb47121127e337a6706d13c5722e96ae9062b"} err="failed to get container status \"428cc127c34e536430a5519d7f3bb47121127e337a6706d13c5722e96ae9062b\": rpc error: code = NotFound desc = could not find container \"428cc127c34e536430a5519d7f3bb47121127e337a6706d13c5722e96ae9062b\": container with ID starting with 428cc127c34e536430a5519d7f3bb47121127e337a6706d13c5722e96ae9062b not found: ID does not exist" Oct 30 00:43:47.985001 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:47.984983 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:43:48.039613 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:48.039586 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cbd5c8c4-fbqjc"] Oct 30 00:43:48.045019 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:48.044993 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cbd5c8c4-fbqjc"] Oct 30 00:43:48.526680 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:48.526651 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-89gxj" Oct 30 00:43:49.817112 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:49.817077 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffa2afe9-832d-452f-855d-982f9e47a601" path="/var/lib/kubelet/pods/ffa2afe9-832d-452f-855d-982f9e47a601/volumes" Oct 30 00:43:51.590900 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:43:51.590866 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:43:57.219889 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:57.219857 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:43:57.412604 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:43:57.412572 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:00.715206 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:00.715173 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:00.792925 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:00.792889 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:03.134522 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:03.134480 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 00:44:03.135051 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:03.134900 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e991db07-d748-4ced-8d7a-4266194e89d8" containerName="prometheus" containerID="cri-o://6641e039709bb8422093c46dee9a083e9985c4918367fb0856c71dbd1215b86e" gracePeriod=600 Oct 30 00:44:03.135051 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:03.134934 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e991db07-d748-4ced-8d7a-4266194e89d8" containerName="thanos-sidecar" containerID="cri-o://2d505ff75a3bcd1c649feb088b62ddd264d0c46069cfb4b7de39b9c367692c1c" gracePeriod=600 Oct 30 00:44:03.135051 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:03.134984 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e991db07-d748-4ced-8d7a-4266194e89d8" containerName="kube-rbac-proxy-thanos" containerID="cri-o://22e127c9764069b72c51267dcfbe5fb0a4aca984d9a03661f64f42c198ec4571" gracePeriod=600 Oct 30 00:44:03.135275 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:03.135051 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e991db07-d748-4ced-8d7a-4266194e89d8" containerName="config-reloader" containerID="cri-o://eb7240966081b8adabf0816e2a664cd069bd5e95134fc0b9e7cf0c3b37bfe767" gracePeriod=600 Oct 30 00:44:03.135275 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:03.134911 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e991db07-d748-4ced-8d7a-4266194e89d8" containerName="kube-rbac-proxy" containerID="cri-o://ef4b00c799ec95588f03c54a29a6477e47df634086638ec09304928e4ccf470f" gracePeriod=600 Oct 30 00:44:03.135275 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:03.135086 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e991db07-d748-4ced-8d7a-4266194e89d8" containerName="kube-rbac-proxy-web" containerID="cri-o://9462859e84a98d3bdbe6ffd54831d6576829ba911f5f0c69fc66527fab52787f" gracePeriod=600 Oct 30 00:44:04.024887 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.024854 5300 generic.go:356] "Generic (PLEG): container finished" podID="e991db07-d748-4ced-8d7a-4266194e89d8" containerID="22e127c9764069b72c51267dcfbe5fb0a4aca984d9a03661f64f42c198ec4571" exitCode=0 Oct 30 00:44:04.024887 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.024879 5300 generic.go:356] "Generic (PLEG): container finished" podID="e991db07-d748-4ced-8d7a-4266194e89d8" containerID="ef4b00c799ec95588f03c54a29a6477e47df634086638ec09304928e4ccf470f" exitCode=0 Oct 30 00:44:04.024887 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.024886 5300 generic.go:356] "Generic (PLEG): container finished" podID="e991db07-d748-4ced-8d7a-4266194e89d8" containerID="2d505ff75a3bcd1c649feb088b62ddd264d0c46069cfb4b7de39b9c367692c1c" exitCode=0 Oct 30 00:44:04.024887 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.024891 5300 generic.go:356] "Generic (PLEG): container finished" podID="e991db07-d748-4ced-8d7a-4266194e89d8" containerID="eb7240966081b8adabf0816e2a664cd069bd5e95134fc0b9e7cf0c3b37bfe767" exitCode=0 Oct 30 00:44:04.024887 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.024896 5300 generic.go:356] "Generic (PLEG): container finished" podID="e991db07-d748-4ced-8d7a-4266194e89d8" containerID="6641e039709bb8422093c46dee9a083e9985c4918367fb0856c71dbd1215b86e" exitCode=0 Oct 30 00:44:04.025176 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.024927 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e991db07-d748-4ced-8d7a-4266194e89d8","Type":"ContainerDied","Data":"22e127c9764069b72c51267dcfbe5fb0a4aca984d9a03661f64f42c198ec4571"} Oct 30 00:44:04.025176 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.024995 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e991db07-d748-4ced-8d7a-4266194e89d8","Type":"ContainerDied","Data":"ef4b00c799ec95588f03c54a29a6477e47df634086638ec09304928e4ccf470f"} Oct 30 00:44:04.025176 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.025021 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e991db07-d748-4ced-8d7a-4266194e89d8","Type":"ContainerDied","Data":"2d505ff75a3bcd1c649feb088b62ddd264d0c46069cfb4b7de39b9c367692c1c"} Oct 30 00:44:04.025176 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.025045 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e991db07-d748-4ced-8d7a-4266194e89d8","Type":"ContainerDied","Data":"eb7240966081b8adabf0816e2a664cd069bd5e95134fc0b9e7cf0c3b37bfe767"} Oct 30 00:44:04.025176 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.025070 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e991db07-d748-4ced-8d7a-4266194e89d8","Type":"ContainerDied","Data":"6641e039709bb8422093c46dee9a083e9985c4918367fb0856c71dbd1215b86e"} Oct 30 00:44:04.376332 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.376308 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:04.489995 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.489966 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-config\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-config\") pod \"e991db07-d748-4ced-8d7a-4266194e89d8\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " Oct 30 00:44:04.490153 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.490007 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-grpc-tls\") pod \"e991db07-d748-4ced-8d7a-4266194e89d8\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " Oct 30 00:44:04.490153 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.490040 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-metrics-client-ca\") pod \"e991db07-d748-4ced-8d7a-4266194e89d8\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " Oct 30 00:44:04.490283 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.490257 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-kube-rbac-proxy\") pod \"e991db07-d748-4ced-8d7a-4266194e89d8\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " Oct 30 00:44:04.490334 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.490314 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"e991db07-d748-4ced-8d7a-4266194e89d8\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " Oct 30 00:44:04.490389 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.490351 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-1\") pod \"e991db07-d748-4ced-8d7a-4266194e89d8\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " Oct 30 00:44:04.490445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.490384 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-kubelet-serving-ca-bundle\") pod \"e991db07-d748-4ced-8d7a-4266194e89d8\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " Oct 30 00:44:04.490445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.490419 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"e991db07-d748-4ced-8d7a-4266194e89d8\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " Oct 30 00:44:04.490445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.490416 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-metrics-client-ca" pod "e991db07-d748-4ced-8d7a-4266194e89d8" (UID: "e991db07-d748-4ced-8d7a-4266194e89d8"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:44:04.490599 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.490448 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-web-config\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-web-config\") pod \"e991db07-d748-4ced-8d7a-4266194e89d8\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " Oct 30 00:44:04.490599 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.490491 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-0\") pod \"e991db07-d748-4ced-8d7a-4266194e89d8\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " Oct 30 00:44:04.490599 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.490539 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e991db07-d748-4ced-8d7a-4266194e89d8-tls-assets\" (UniqueName: \"kubernetes.io/projected/e991db07-d748-4ced-8d7a-4266194e89d8-tls-assets\") pod \"e991db07-d748-4ced-8d7a-4266194e89d8\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " Oct 30 00:44:04.490599 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.490574 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-metrics-client-certs\") pod \"e991db07-d748-4ced-8d7a-4266194e89d8\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " Oct 30 00:44:04.490818 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.490607 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-trusted-ca-bundle\") pod \"e991db07-d748-4ced-8d7a-4266194e89d8\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " Oct 30 00:44:04.490818 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.490635 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-tls\") pod \"e991db07-d748-4ced-8d7a-4266194e89d8\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " Oct 30 00:44:04.490818 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.490668 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e991db07-d748-4ced-8d7a-4266194e89d8-config-out\" (UniqueName: \"kubernetes.io/empty-dir/e991db07-d748-4ced-8d7a-4266194e89d8-config-out\") pod \"e991db07-d748-4ced-8d7a-4266194e89d8\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " Oct 30 00:44:04.490818 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.490700 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-db\") pod \"e991db07-d748-4ced-8d7a-4266194e89d8\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " Oct 30 00:44:04.490818 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.490789 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-1" pod "e991db07-d748-4ced-8d7a-4266194e89d8" (UID: "e991db07-d748-4ced-8d7a-4266194e89d8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:44:04.491341 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.491179 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-kubelet-serving-ca-bundle" pod "e991db07-d748-4ced-8d7a-4266194e89d8" (UID: "e991db07-d748-4ced-8d7a-4266194e89d8"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:44:04.491341 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.491293 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-trusted-ca-bundle" pod "e991db07-d748-4ced-8d7a-4266194e89d8" (UID: "e991db07-d748-4ced-8d7a-4266194e89d8"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:44:04.493206 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.492892 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-grpc-tls" pod "e991db07-d748-4ced-8d7a-4266194e89d8" (UID: "e991db07-d748-4ced-8d7a-4266194e89d8"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:44:04.493206 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.492957 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-db" pod "e991db07-d748-4ced-8d7a-4266194e89d8" (UID: "e991db07-d748-4ced-8d7a-4266194e89d8"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:44:04.493699 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.493672 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-0" pod "e991db07-d748-4ced-8d7a-4266194e89d8" (UID: "e991db07-d748-4ced-8d7a-4266194e89d8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:44:04.493815 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.493740 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-2\") pod \"e991db07-d748-4ced-8d7a-4266194e89d8\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " Oct 30 00:44:04.493815 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.493790 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-serving-certs-ca-bundle\") pod \"e991db07-d748-4ced-8d7a-4266194e89d8\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " Oct 30 00:44:04.493936 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.493827 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-thanos-prometheus-http-client-file\") pod \"e991db07-d748-4ced-8d7a-4266194e89d8\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " Oct 30 00:44:04.493936 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.493861 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e991db07-d748-4ced-8d7a-4266194e89d8-kube-api-access-mghhs\" (UniqueName: \"kubernetes.io/projected/e991db07-d748-4ced-8d7a-4266194e89d8-kube-api-access-mghhs\") pod \"e991db07-d748-4ced-8d7a-4266194e89d8\" (UID: \"e991db07-d748-4ced-8d7a-4266194e89d8\") " Oct 30 00:44:04.494034 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.493943 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-thanos-sidecar-tls" pod "e991db07-d748-4ced-8d7a-4266194e89d8" (UID: "e991db07-d748-4ced-8d7a-4266194e89d8"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:44:04.494220 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.494132 5300 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:04.494683 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.494660 5300 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-db\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:04.494844 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.494687 5300 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-grpc-tls\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:04.494844 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.494716 5300 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-metrics-client-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:04.494844 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.494734 5300 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:04.495326 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.495286 5300 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:04.495326 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.495313 5300 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:04.495478 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.495328 5300 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:04.495478 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.494308 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-serving-certs-ca-bundle" pod "e991db07-d748-4ced-8d7a-4266194e89d8" (UID: "e991db07-d748-4ced-8d7a-4266194e89d8"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:44:04.495478 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.494612 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-2" pod "e991db07-d748-4ced-8d7a-4266194e89d8" (UID: "e991db07-d748-4ced-8d7a-4266194e89d8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:44:04.495478 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.494994 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e991db07-d748-4ced-8d7a-4266194e89d8-tls-assets" pod "e991db07-d748-4ced-8d7a-4266194e89d8" (UID: "e991db07-d748-4ced-8d7a-4266194e89d8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:44:04.495745 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.495492 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-tls" pod "e991db07-d748-4ced-8d7a-4266194e89d8" (UID: "e991db07-d748-4ced-8d7a-4266194e89d8"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:44:04.495745 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.495606 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-kube-rbac-proxy" pod "e991db07-d748-4ced-8d7a-4266194e89d8" (UID: "e991db07-d748-4ced-8d7a-4266194e89d8"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:44:04.495745 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.495680 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-config" pod "e991db07-d748-4ced-8d7a-4266194e89d8" (UID: "e991db07-d748-4ced-8d7a-4266194e89d8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:44:04.495929 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.495774 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-metrics-client-certs" pod "e991db07-d748-4ced-8d7a-4266194e89d8" (UID: "e991db07-d748-4ced-8d7a-4266194e89d8"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:44:04.496599 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.496560 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-kube-rbac-proxy-web" pod "e991db07-d748-4ced-8d7a-4266194e89d8" (UID: "e991db07-d748-4ced-8d7a-4266194e89d8"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:44:04.497028 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.497005 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e991db07-d748-4ced-8d7a-4266194e89d8-config-out" pod "e991db07-d748-4ced-8d7a-4266194e89d8" (UID: "e991db07-d748-4ced-8d7a-4266194e89d8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:44:04.497442 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.497422 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e991db07-d748-4ced-8d7a-4266194e89d8-kube-api-access-mghhs" pod "e991db07-d748-4ced-8d7a-4266194e89d8" (UID: "e991db07-d748-4ced-8d7a-4266194e89d8"). InnerVolumeSpecName "kube-api-access-mghhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:44:04.498244 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.498213 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-thanos-prometheus-http-client-file" pod "e991db07-d748-4ced-8d7a-4266194e89d8" (UID: "e991db07-d748-4ced-8d7a-4266194e89d8"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:44:04.505889 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.505851 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-web-config" pod "e991db07-d748-4ced-8d7a-4266194e89d8" (UID: "e991db07-d748-4ced-8d7a-4266194e89d8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:44:04.596266 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.596190 5300 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e991db07-d748-4ced-8d7a-4266194e89d8-tls-assets\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:04.596266 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.596220 5300 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-metrics-client-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:04.596266 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.596231 5300 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:04.596266 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.596240 5300 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e991db07-d748-4ced-8d7a-4266194e89d8-config-out\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:04.596266 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.596249 5300 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:04.596266 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.596258 5300 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e991db07-d748-4ced-8d7a-4266194e89d8-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:04.596266 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.596267 5300 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:04.596587 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.596277 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mghhs\" (UniqueName: \"kubernetes.io/projected/e991db07-d748-4ced-8d7a-4266194e89d8-kube-api-access-mghhs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:04.596587 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.596286 5300 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-config\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:04.596587 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.596295 5300 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-kube-rbac-proxy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:04.596587 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.596304 5300 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:04.596587 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:04.596313 5300 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e991db07-d748-4ced-8d7a-4266194e89d8-web-config\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:05.035224 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.035194 5300 generic.go:356] "Generic (PLEG): container finished" podID="e991db07-d748-4ced-8d7a-4266194e89d8" containerID="9462859e84a98d3bdbe6ffd54831d6576829ba911f5f0c69fc66527fab52787f" exitCode=0 Oct 30 00:44:05.035399 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.035275 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e991db07-d748-4ced-8d7a-4266194e89d8","Type":"ContainerDied","Data":"9462859e84a98d3bdbe6ffd54831d6576829ba911f5f0c69fc66527fab52787f"} Oct 30 00:44:05.035399 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.035309 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.035399 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.035383 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e991db07-d748-4ced-8d7a-4266194e89d8","Type":"ContainerDied","Data":"cc6e866c85861f00cee7ad73ec83efa452092d9d609585d1e4be909d96416e2c"} Oct 30 00:44:05.035582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.035432 5300 scope.go:122] "RemoveContainer" containerID="22e127c9764069b72c51267dcfbe5fb0a4aca984d9a03661f64f42c198ec4571" Oct 30 00:44:05.044900 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.043807 5300 scope.go:122] "RemoveContainer" containerID="ef4b00c799ec95588f03c54a29a6477e47df634086638ec09304928e4ccf470f" Oct 30 00:44:05.051951 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.051934 5300 scope.go:122] "RemoveContainer" containerID="9462859e84a98d3bdbe6ffd54831d6576829ba911f5f0c69fc66527fab52787f" Oct 30 00:44:05.057980 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.057963 5300 scope.go:122] "RemoveContainer" containerID="2d505ff75a3bcd1c649feb088b62ddd264d0c46069cfb4b7de39b9c367692c1c" Oct 30 00:44:05.064059 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.064046 5300 scope.go:122] "RemoveContainer" containerID="eb7240966081b8adabf0816e2a664cd069bd5e95134fc0b9e7cf0c3b37bfe767" Oct 30 00:44:05.070120 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.070105 5300 scope.go:122] "RemoveContainer" containerID="6641e039709bb8422093c46dee9a083e9985c4918367fb0856c71dbd1215b86e" Oct 30 00:44:05.076625 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.076609 5300 scope.go:122] "RemoveContainer" containerID="4cde129fff51071d51c81973829d7b60cbdc27630f79782f7aa3fcb590cfe88f" Oct 30 00:44:05.082980 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.082964 5300 scope.go:122] "RemoveContainer" containerID="22e127c9764069b72c51267dcfbe5fb0a4aca984d9a03661f64f42c198ec4571" Oct 30 00:44:05.083219 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:44:05.083200 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22e127c9764069b72c51267dcfbe5fb0a4aca984d9a03661f64f42c198ec4571\": container with ID starting with 22e127c9764069b72c51267dcfbe5fb0a4aca984d9a03661f64f42c198ec4571 not found: ID does not exist" containerID="22e127c9764069b72c51267dcfbe5fb0a4aca984d9a03661f64f42c198ec4571" Oct 30 00:44:05.083270 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.083229 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22e127c9764069b72c51267dcfbe5fb0a4aca984d9a03661f64f42c198ec4571"} err="failed to get container status \"22e127c9764069b72c51267dcfbe5fb0a4aca984d9a03661f64f42c198ec4571\": rpc error: code = NotFound desc = could not find container \"22e127c9764069b72c51267dcfbe5fb0a4aca984d9a03661f64f42c198ec4571\": container with ID starting with 22e127c9764069b72c51267dcfbe5fb0a4aca984d9a03661f64f42c198ec4571 not found: ID does not exist" Oct 30 00:44:05.083270 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.083246 5300 scope.go:122] "RemoveContainer" containerID="ef4b00c799ec95588f03c54a29a6477e47df634086638ec09304928e4ccf470f" Oct 30 00:44:05.083439 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:44:05.083422 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef4b00c799ec95588f03c54a29a6477e47df634086638ec09304928e4ccf470f\": container with ID starting with ef4b00c799ec95588f03c54a29a6477e47df634086638ec09304928e4ccf470f not found: ID does not exist" containerID="ef4b00c799ec95588f03c54a29a6477e47df634086638ec09304928e4ccf470f" Oct 30 00:44:05.083487 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.083445 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef4b00c799ec95588f03c54a29a6477e47df634086638ec09304928e4ccf470f"} err="failed to get container status \"ef4b00c799ec95588f03c54a29a6477e47df634086638ec09304928e4ccf470f\": rpc error: code = NotFound desc = could not find container \"ef4b00c799ec95588f03c54a29a6477e47df634086638ec09304928e4ccf470f\": container with ID starting with ef4b00c799ec95588f03c54a29a6477e47df634086638ec09304928e4ccf470f not found: ID does not exist" Oct 30 00:44:05.083487 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.083462 5300 scope.go:122] "RemoveContainer" containerID="9462859e84a98d3bdbe6ffd54831d6576829ba911f5f0c69fc66527fab52787f" Oct 30 00:44:05.083664 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:44:05.083650 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9462859e84a98d3bdbe6ffd54831d6576829ba911f5f0c69fc66527fab52787f\": container with ID starting with 9462859e84a98d3bdbe6ffd54831d6576829ba911f5f0c69fc66527fab52787f not found: ID does not exist" containerID="9462859e84a98d3bdbe6ffd54831d6576829ba911f5f0c69fc66527fab52787f" Oct 30 00:44:05.083706 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.083668 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9462859e84a98d3bdbe6ffd54831d6576829ba911f5f0c69fc66527fab52787f"} err="failed to get container status \"9462859e84a98d3bdbe6ffd54831d6576829ba911f5f0c69fc66527fab52787f\": rpc error: code = NotFound desc = could not find container \"9462859e84a98d3bdbe6ffd54831d6576829ba911f5f0c69fc66527fab52787f\": container with ID starting with 9462859e84a98d3bdbe6ffd54831d6576829ba911f5f0c69fc66527fab52787f not found: ID does not exist" Oct 30 00:44:05.083706 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.083680 5300 scope.go:122] "RemoveContainer" containerID="2d505ff75a3bcd1c649feb088b62ddd264d0c46069cfb4b7de39b9c367692c1c" Oct 30 00:44:05.083930 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:44:05.083914 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d505ff75a3bcd1c649feb088b62ddd264d0c46069cfb4b7de39b9c367692c1c\": container with ID starting with 2d505ff75a3bcd1c649feb088b62ddd264d0c46069cfb4b7de39b9c367692c1c not found: ID does not exist" containerID="2d505ff75a3bcd1c649feb088b62ddd264d0c46069cfb4b7de39b9c367692c1c" Oct 30 00:44:05.083972 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.083935 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d505ff75a3bcd1c649feb088b62ddd264d0c46069cfb4b7de39b9c367692c1c"} err="failed to get container status \"2d505ff75a3bcd1c649feb088b62ddd264d0c46069cfb4b7de39b9c367692c1c\": rpc error: code = NotFound desc = could not find container \"2d505ff75a3bcd1c649feb088b62ddd264d0c46069cfb4b7de39b9c367692c1c\": container with ID starting with 2d505ff75a3bcd1c649feb088b62ddd264d0c46069cfb4b7de39b9c367692c1c not found: ID does not exist" Oct 30 00:44:05.083972 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.083948 5300 scope.go:122] "RemoveContainer" containerID="eb7240966081b8adabf0816e2a664cd069bd5e95134fc0b9e7cf0c3b37bfe767" Oct 30 00:44:05.084150 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:44:05.084137 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb7240966081b8adabf0816e2a664cd069bd5e95134fc0b9e7cf0c3b37bfe767\": container with ID starting with eb7240966081b8adabf0816e2a664cd069bd5e95134fc0b9e7cf0c3b37bfe767 not found: ID does not exist" containerID="eb7240966081b8adabf0816e2a664cd069bd5e95134fc0b9e7cf0c3b37bfe767" Oct 30 00:44:05.084193 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.084153 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb7240966081b8adabf0816e2a664cd069bd5e95134fc0b9e7cf0c3b37bfe767"} err="failed to get container status \"eb7240966081b8adabf0816e2a664cd069bd5e95134fc0b9e7cf0c3b37bfe767\": rpc error: code = NotFound desc = could not find container \"eb7240966081b8adabf0816e2a664cd069bd5e95134fc0b9e7cf0c3b37bfe767\": container with ID starting with eb7240966081b8adabf0816e2a664cd069bd5e95134fc0b9e7cf0c3b37bfe767 not found: ID does not exist" Oct 30 00:44:05.084193 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.084166 5300 scope.go:122] "RemoveContainer" containerID="6641e039709bb8422093c46dee9a083e9985c4918367fb0856c71dbd1215b86e" Oct 30 00:44:05.084351 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:44:05.084337 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6641e039709bb8422093c46dee9a083e9985c4918367fb0856c71dbd1215b86e\": container with ID starting with 6641e039709bb8422093c46dee9a083e9985c4918367fb0856c71dbd1215b86e not found: ID does not exist" containerID="6641e039709bb8422093c46dee9a083e9985c4918367fb0856c71dbd1215b86e" Oct 30 00:44:05.084390 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.084354 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6641e039709bb8422093c46dee9a083e9985c4918367fb0856c71dbd1215b86e"} err="failed to get container status \"6641e039709bb8422093c46dee9a083e9985c4918367fb0856c71dbd1215b86e\": rpc error: code = NotFound desc = could not find container \"6641e039709bb8422093c46dee9a083e9985c4918367fb0856c71dbd1215b86e\": container with ID starting with 6641e039709bb8422093c46dee9a083e9985c4918367fb0856c71dbd1215b86e not found: ID does not exist" Oct 30 00:44:05.084390 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.084365 5300 scope.go:122] "RemoveContainer" containerID="4cde129fff51071d51c81973829d7b60cbdc27630f79782f7aa3fcb590cfe88f" Oct 30 00:44:05.084562 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:44:05.084545 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cde129fff51071d51c81973829d7b60cbdc27630f79782f7aa3fcb590cfe88f\": container with ID starting with 4cde129fff51071d51c81973829d7b60cbdc27630f79782f7aa3fcb590cfe88f not found: ID does not exist" containerID="4cde129fff51071d51c81973829d7b60cbdc27630f79782f7aa3fcb590cfe88f" Oct 30 00:44:05.084602 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.084567 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cde129fff51071d51c81973829d7b60cbdc27630f79782f7aa3fcb590cfe88f"} err="failed to get container status \"4cde129fff51071d51c81973829d7b60cbdc27630f79782f7aa3fcb590cfe88f\": rpc error: code = NotFound desc = could not find container \"4cde129fff51071d51c81973829d7b60cbdc27630f79782f7aa3fcb590cfe88f\": container with ID starting with 4cde129fff51071d51c81973829d7b60cbdc27630f79782f7aa3fcb590cfe88f not found: ID does not exist" Oct 30 00:44:05.104456 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.104432 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 00:44:05.117844 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.117823 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 00:44:05.132556 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.132533 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 00:44:05.132803 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.132787 5300 cpu_manager.go:414] "containerMap: removing container" podUID="e991db07-d748-4ced-8d7a-4266194e89d8" containerName="thanos-sidecar" Oct 30 00:44:05.132861 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.132805 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e991db07-d748-4ced-8d7a-4266194e89d8" containerName="thanos-sidecar" Oct 30 00:44:05.132861 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.132817 5300 cpu_manager.go:414] "containerMap: removing container" podUID="ffa2afe9-832d-452f-855d-982f9e47a601" containerName="console" Oct 30 00:44:05.132861 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.132822 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffa2afe9-832d-452f-855d-982f9e47a601" containerName="console" Oct 30 00:44:05.132861 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.132835 5300 cpu_manager.go:414] "containerMap: removing container" podUID="e991db07-d748-4ced-8d7a-4266194e89d8" containerName="kube-rbac-proxy-web" Oct 30 00:44:05.132861 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.132840 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e991db07-d748-4ced-8d7a-4266194e89d8" containerName="kube-rbac-proxy-web" Oct 30 00:44:05.132861 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.132846 5300 cpu_manager.go:414] "containerMap: removing container" podUID="e991db07-d748-4ced-8d7a-4266194e89d8" containerName="kube-rbac-proxy" Oct 30 00:44:05.132861 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.132850 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e991db07-d748-4ced-8d7a-4266194e89d8" containerName="kube-rbac-proxy" Oct 30 00:44:05.132861 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.132856 5300 cpu_manager.go:414] "containerMap: removing container" podUID="e991db07-d748-4ced-8d7a-4266194e89d8" containerName="kube-rbac-proxy-thanos" Oct 30 00:44:05.132861 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.132861 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e991db07-d748-4ced-8d7a-4266194e89d8" containerName="kube-rbac-proxy-thanos" Oct 30 00:44:05.133129 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.132868 5300 cpu_manager.go:414] "containerMap: removing container" podUID="e991db07-d748-4ced-8d7a-4266194e89d8" containerName="prometheus" Oct 30 00:44:05.133129 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.132873 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e991db07-d748-4ced-8d7a-4266194e89d8" containerName="prometheus" Oct 30 00:44:05.133129 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.132883 5300 cpu_manager.go:414] "containerMap: removing container" podUID="e991db07-d748-4ced-8d7a-4266194e89d8" containerName="config-reloader" Oct 30 00:44:05.133129 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.132890 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e991db07-d748-4ced-8d7a-4266194e89d8" containerName="config-reloader" Oct 30 00:44:05.133129 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.132896 5300 cpu_manager.go:414] "containerMap: removing container" podUID="e991db07-d748-4ced-8d7a-4266194e89d8" containerName="init-config-reloader" Oct 30 00:44:05.133129 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.132900 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e991db07-d748-4ced-8d7a-4266194e89d8" containerName="init-config-reloader" Oct 30 00:44:05.133129 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.132951 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="e991db07-d748-4ced-8d7a-4266194e89d8" containerName="thanos-sidecar" Oct 30 00:44:05.133129 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.132960 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffa2afe9-832d-452f-855d-982f9e47a601" containerName="console" Oct 30 00:44:05.133129 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.132969 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="e991db07-d748-4ced-8d7a-4266194e89d8" containerName="kube-rbac-proxy-web" Oct 30 00:44:05.133129 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.132974 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="e991db07-d748-4ced-8d7a-4266194e89d8" containerName="kube-rbac-proxy" Oct 30 00:44:05.133129 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.132978 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="e991db07-d748-4ced-8d7a-4266194e89d8" containerName="kube-rbac-proxy-thanos" Oct 30 00:44:05.133129 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.132984 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="e991db07-d748-4ced-8d7a-4266194e89d8" containerName="prometheus" Oct 30 00:44:05.133129 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.132988 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="e991db07-d748-4ced-8d7a-4266194e89d8" containerName="config-reloader" Oct 30 00:44:05.138373 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.138357 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.154416 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.154393 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 00:44:05.154627 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.154612 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 00:44:05.154710 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.154614 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 00:44:05.154946 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.154930 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 00:44:05.155037 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.154960 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 00:44:05.155037 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.155018 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 00:44:05.155154 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.155083 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 00:44:05.155209 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.155153 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 00:44:05.155209 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.155170 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 00:44:05.155323 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.155304 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 00:44:05.155382 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.155321 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 00:44:05.168145 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.167935 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8uop28j2hbr7f\"" Oct 30 00:44:05.168145 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.168043 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dzgqx\"" Oct 30 00:44:05.168145 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.168091 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 00:44:05.168389 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.168153 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 00:44:05.169026 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.169004 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 00:44:05.169266 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.169245 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 00:44:05.169932 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.169910 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 00:44:05.202039 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.202004 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.202240 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.202044 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s65zm\" (UniqueName: \"kubernetes.io/projected/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-kube-api-access-s65zm\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.202240 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.202072 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-config-out\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.202240 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.202134 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.202240 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.202183 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.202240 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.202213 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.202394 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.202301 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.202394 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.202334 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.202394 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.202356 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.202483 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.202443 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-web-config\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.202483 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.202470 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-config\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.202544 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.202489 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.202544 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.202521 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.202608 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.202565 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.202608 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.202598 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.202668 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.202622 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.202668 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.202641 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.202668 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.202663 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.202769 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.202694 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.202769 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.202724 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.303596 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.303507 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-web-config\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.303596 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.303549 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-config\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.303596 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.303573 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.303968 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.303600 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.303968 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.303635 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.303968 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.303659 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.303968 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.303689 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.303968 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.303713 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.303968 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.303742 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.303968 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.303798 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.303968 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.303823 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.303968 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.303849 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.303968 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.303884 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s65zm\" (UniqueName: \"kubernetes.io/projected/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-kube-api-access-s65zm\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.303968 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.303910 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-config-out\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.303968 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.303933 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.303968 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.303958 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.304597 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.303985 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.304597 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.304018 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.304597 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.304049 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.304597 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.304081 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.304597 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.304129 5300 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 00:44:05.304597 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.304211 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.304597 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.304388 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.304597 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.304506 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.305044 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.304600 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.305044 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.305030 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.307163 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.306950 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.307163 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.307109 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-web-config\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.307360 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.307166 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.307360 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.307273 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.307474 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.307385 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-config\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.307638 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.307610 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.307845 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.307822 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.308060 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.308035 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.308485 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.308461 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.309330 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.309310 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.309478 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.309450 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.309919 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.309891 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.309994 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.309979 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.310029 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.309988 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-config-out\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.324858 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.324830 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s65zm\" (UniqueName: \"kubernetes.io/projected/b85f5ff6-7882-40e1-a28a-1cb8cf13c5be-kube-api-access-s65zm\") pod \"prometheus-k8s-0\" (UID: \"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.472145 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.472109 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:05.676212 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.676133 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-76d5554bbf-5dg2c"] Oct 30 00:44:05.679425 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.679408 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:05.702371 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.702344 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76d5554bbf-5dg2c"] Oct 30 00:44:05.791771 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.791717 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:05.793080 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.793051 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 00:44:05.795958 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:44:05.795929 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85f5ff6_7882_40e1_a28a_1cb8cf13c5be.slice/crio-c7f4e479e47821018c7717921d012f53029354f8d089d282c2d0d1104efcd512 WatchSource:0}: Error finding container c7f4e479e47821018c7717921d012f53029354f8d089d282c2d0d1104efcd512: Status 404 returned error can't find the container with id c7f4e479e47821018c7717921d012f53029354f8d089d282c2d0d1104efcd512 Oct 30 00:44:05.809230 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.809210 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/202bf6a4-7b22-4988-893f-ca280f78fca5-console-oauth-config\") pod \"console-76d5554bbf-5dg2c\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:05.809290 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.809242 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-service-ca\") pod \"console-76d5554bbf-5dg2c\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:05.809290 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.809265 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-oauth-serving-cert\") pod \"console-76d5554bbf-5dg2c\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:05.809413 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.809397 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/202bf6a4-7b22-4988-893f-ca280f78fca5-tmp\") pod \"console-76d5554bbf-5dg2c\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:05.809476 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.809435 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-trusted-ca-bundle\") pod \"console-76d5554bbf-5dg2c\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:05.809530 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.809498 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/202bf6a4-7b22-4988-893f-ca280f78fca5-console-serving-cert\") pod \"console-76d5554bbf-5dg2c\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:05.809573 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.809547 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-console-config\") pod \"console-76d5554bbf-5dg2c\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:05.809573 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.809567 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgvr7\" (UniqueName: \"kubernetes.io/projected/202bf6a4-7b22-4988-893f-ca280f78fca5-kube-api-access-fgvr7\") pod \"console-76d5554bbf-5dg2c\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:05.817725 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.817701 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e991db07-d748-4ced-8d7a-4266194e89d8" path="/var/lib/kubelet/pods/e991db07-d748-4ced-8d7a-4266194e89d8/volumes" Oct 30 00:44:05.888740 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.888703 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:05.910436 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.910417 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/202bf6a4-7b22-4988-893f-ca280f78fca5-tmp\") pod \"console-76d5554bbf-5dg2c\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:05.910536 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.910447 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-trusted-ca-bundle\") pod \"console-76d5554bbf-5dg2c\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:05.910536 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.910471 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/202bf6a4-7b22-4988-893f-ca280f78fca5-console-serving-cert\") pod \"console-76d5554bbf-5dg2c\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:05.910536 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.910492 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-console-config\") pod \"console-76d5554bbf-5dg2c\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:05.910536 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.910515 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgvr7\" (UniqueName: \"kubernetes.io/projected/202bf6a4-7b22-4988-893f-ca280f78fca5-kube-api-access-fgvr7\") pod \"console-76d5554bbf-5dg2c\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:05.910697 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.910564 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/202bf6a4-7b22-4988-893f-ca280f78fca5-console-oauth-config\") pod \"console-76d5554bbf-5dg2c\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:05.910697 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.910587 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-service-ca\") pod \"console-76d5554bbf-5dg2c\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:05.910697 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.910614 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-oauth-serving-cert\") pod \"console-76d5554bbf-5dg2c\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:05.911043 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.910996 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/202bf6a4-7b22-4988-893f-ca280f78fca5-tmp\") pod \"console-76d5554bbf-5dg2c\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:05.911412 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.911389 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-service-ca\") pod \"console-76d5554bbf-5dg2c\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:05.911412 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.911404 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-oauth-serving-cert\") pod \"console-76d5554bbf-5dg2c\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:05.911570 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.911392 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-console-config\") pod \"console-76d5554bbf-5dg2c\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:05.911623 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.911600 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-trusted-ca-bundle\") pod \"console-76d5554bbf-5dg2c\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:05.912798 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.912777 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/202bf6a4-7b22-4988-893f-ca280f78fca5-console-oauth-config\") pod \"console-76d5554bbf-5dg2c\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:05.912912 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.912898 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/202bf6a4-7b22-4988-893f-ca280f78fca5-console-serving-cert\") pod \"console-76d5554bbf-5dg2c\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:05.930140 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.930085 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgvr7\" (UniqueName: \"kubernetes.io/projected/202bf6a4-7b22-4988-893f-ca280f78fca5-kube-api-access-fgvr7\") pod \"console-76d5554bbf-5dg2c\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:05.990242 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:05.990205 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:06.052395 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:06.049770 5300 generic.go:356] "Generic (PLEG): container finished" podID="b85f5ff6-7882-40e1-a28a-1cb8cf13c5be" containerID="cc1e29f23c584d55722402d43fb437250e82a4a215cfe819b706702300a5e16a" exitCode=0 Oct 30 00:44:06.052395 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:06.049800 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be","Type":"ContainerDied","Data":"cc1e29f23c584d55722402d43fb437250e82a4a215cfe819b706702300a5e16a"} Oct 30 00:44:06.052395 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:06.049902 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be","Type":"ContainerStarted","Data":"c7f4e479e47821018c7717921d012f53029354f8d089d282c2d0d1104efcd512"} Oct 30 00:44:06.102432 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:06.102403 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:06.203445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:06.203411 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:06.290905 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:06.290880 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76d5554bbf-5dg2c"] Oct 30 00:44:07.058351 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:07.058318 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be","Type":"ContainerStarted","Data":"b4f4b85f159b79e1e01c663c068d810605e70d9350fab0729561806fd26e8bec"} Oct 30 00:44:07.058807 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:07.058388 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be","Type":"ContainerStarted","Data":"c59573a31f7bf6faa7727c7e624dbdb102e2b1426c9633e06c56a429f120549b"} Oct 30 00:44:07.058807 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:07.058432 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be","Type":"ContainerStarted","Data":"ba111b239497bba253b3b70d21d09183dee801bd37e7d7fc9e67178875da5df1"} Oct 30 00:44:07.058807 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:07.058471 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be","Type":"ContainerStarted","Data":"af5960c94714fae5686bbe86044fd7ad3eeb9a80b17bea675d3e352a2e61c7f7"} Oct 30 00:44:07.058807 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:07.058509 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be","Type":"ContainerStarted","Data":"162a4a48e6d0e0f9c8aefb834ab5d3dac8cf401f99511b0fd9aa69d73e21bd49"} Oct 30 00:44:07.058807 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:07.058548 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b85f5ff6-7882-40e1-a28a-1cb8cf13c5be","Type":"ContainerStarted","Data":"5d8b57acd7e6ad5006af99c721dc650375cdf4add7c3e989ea1f0dadab7a5a97"} Oct 30 00:44:07.059928 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:07.059897 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76d5554bbf-5dg2c" event={"ID":"202bf6a4-7b22-4988-893f-ca280f78fca5","Type":"ContainerStarted","Data":"cd26ea187c8957717b7c7cf28807617fb0dbeaab274f0cab1aa034dc85c1f145"} Oct 30 00:44:07.060031 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:07.059941 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76d5554bbf-5dg2c" event={"ID":"202bf6a4-7b22-4988-893f-ca280f78fca5","Type":"ContainerStarted","Data":"1fedba3b88ff500a1f96803731a39ea057a0e8f3f3a0c30dea8a31436bade1c2"} Oct 30 00:44:07.162828 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:07.162752 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-76d5554bbf-5dg2c" podStartSLOduration=2.162738154 podStartE2EDuration="2.162738154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:07.162244467 +0000 UTC m=+47.478112791" watchObservedRunningTime="2026-10-30 00:44:07.162738154 +0000 UTC m=+47.478606479" Oct 30 00:44:07.163335 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:07.163293 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.146510766 podStartE2EDuration="2.163282612s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.771846ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:07.121867167 +0000 UTC m=+47.437735508" watchObservedRunningTime="2026-10-30 00:44:07.163282612 +0000 UTC m=+47.479150934" Oct 30 00:44:10.472385 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:10.472351 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:15.990652 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:15.990609 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:15.991072 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:15.990668 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:15.995470 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:15.995443 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:16.095142 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:16.095114 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:44:16.161659 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:16.161630 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8698cc7897-jflvm"] Oct 30 00:44:21.592234 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:44:21.592191 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:44:28.250863 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:28.250827 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:36.394728 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:36.394697 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:41.181118 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:41.181063 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-8698cc7897-jflvm" podUID="07e33ac0-f034-477b-8e5a-aaf82a23d7fa" containerName="console" containerID="cri-o://5f7166b9f2cffea47388c8a6a93e4cb2979ecf3e0663fa7fc6d1740de0ef55d6" gracePeriod=15 Oct 30 00:44:41.419254 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:41.419230 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:44:41.488050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:41.488018 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-serving-cert\") pod \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " Oct 30 00:44:41.488220 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:41.488061 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-kube-api-access-2726g\" (UniqueName: \"kubernetes.io/projected/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-kube-api-access-2726g\") pod \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " Oct 30 00:44:41.488220 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:41.488090 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-oauth-serving-cert\") pod \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " Oct 30 00:44:41.488220 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:41.488139 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-trusted-ca-bundle\") pod \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " Oct 30 00:44:41.488220 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:41.488169 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-oauth-config\") pod \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " Oct 30 00:44:41.488220 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:41.488201 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-config\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-config\") pod \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " Oct 30 00:44:41.488467 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:41.488230 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-service-ca\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-service-ca\") pod \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " Oct 30 00:44:41.488467 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:41.488260 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-tmp\" (UniqueName: \"kubernetes.io/empty-dir/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-tmp\") pod \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\" (UID: \"07e33ac0-f034-477b-8e5a-aaf82a23d7fa\") " Oct 30 00:44:41.488642 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:41.488608 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-oauth-serving-cert" pod "07e33ac0-f034-477b-8e5a-aaf82a23d7fa" (UID: "07e33ac0-f034-477b-8e5a-aaf82a23d7fa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:44:41.488725 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:41.488650 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-trusted-ca-bundle" pod "07e33ac0-f034-477b-8e5a-aaf82a23d7fa" (UID: "07e33ac0-f034-477b-8e5a-aaf82a23d7fa"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:44:41.488802 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:41.488728 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-tmp" pod "07e33ac0-f034-477b-8e5a-aaf82a23d7fa" (UID: "07e33ac0-f034-477b-8e5a-aaf82a23d7fa"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:44:41.488802 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:41.488738 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-service-ca" pod "07e33ac0-f034-477b-8e5a-aaf82a23d7fa" (UID: "07e33ac0-f034-477b-8e5a-aaf82a23d7fa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:44:41.489063 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:41.489032 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-config" pod "07e33ac0-f034-477b-8e5a-aaf82a23d7fa" (UID: "07e33ac0-f034-477b-8e5a-aaf82a23d7fa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:44:41.490394 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:41.490368 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-oauth-config" pod "07e33ac0-f034-477b-8e5a-aaf82a23d7fa" (UID: "07e33ac0-f034-477b-8e5a-aaf82a23d7fa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:44:41.490490 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:41.490467 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-serving-cert" pod "07e33ac0-f034-477b-8e5a-aaf82a23d7fa" (UID: "07e33ac0-f034-477b-8e5a-aaf82a23d7fa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:44:41.490531 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:41.490467 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-kube-api-access-2726g" pod "07e33ac0-f034-477b-8e5a-aaf82a23d7fa" (UID: "07e33ac0-f034-477b-8e5a-aaf82a23d7fa"). InnerVolumeSpecName "kube-api-access-2726g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:44:41.588954 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:41.588917 5300 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-serving-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:41.588954 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:41.588947 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2726g\" (UniqueName: \"kubernetes.io/projected/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-kube-api-access-2726g\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:41.588954 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:41.588957 5300 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-oauth-serving-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:41.589183 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:41.588966 5300 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-trusted-ca-bundle\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:41.589183 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:41.588975 5300 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-oauth-config\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:41.589183 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:41.588984 5300 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-console-config\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:41.589183 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:41.588992 5300 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:41.589183 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:41.589001 5300 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07e33ac0-f034-477b-8e5a-aaf82a23d7fa-tmp\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:44:42.176795 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:42.176690 5300 generic.go:356] "Generic (PLEG): container finished" podID="07e33ac0-f034-477b-8e5a-aaf82a23d7fa" containerID="5f7166b9f2cffea47388c8a6a93e4cb2979ecf3e0663fa7fc6d1740de0ef55d6" exitCode=0 Oct 30 00:44:42.176795 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:42.176744 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8698cc7897-jflvm" Oct 30 00:44:42.176979 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:42.176797 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8698cc7897-jflvm" event={"ID":"07e33ac0-f034-477b-8e5a-aaf82a23d7fa","Type":"ContainerDied","Data":"5f7166b9f2cffea47388c8a6a93e4cb2979ecf3e0663fa7fc6d1740de0ef55d6"} Oct 30 00:44:42.176979 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:42.176857 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8698cc7897-jflvm" event={"ID":"07e33ac0-f034-477b-8e5a-aaf82a23d7fa","Type":"ContainerDied","Data":"a39b78a775f0f6b5a51dd6831da2352abac4740bed387f3af7c58b4c2ef8676c"} Oct 30 00:44:42.176979 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:42.176878 5300 scope.go:122] "RemoveContainer" containerID="5f7166b9f2cffea47388c8a6a93e4cb2979ecf3e0663fa7fc6d1740de0ef55d6" Oct 30 00:44:42.184844 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:42.184696 5300 scope.go:122] "RemoveContainer" containerID="5f7166b9f2cffea47388c8a6a93e4cb2979ecf3e0663fa7fc6d1740de0ef55d6" Oct 30 00:44:42.185094 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:44:42.184989 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f7166b9f2cffea47388c8a6a93e4cb2979ecf3e0663fa7fc6d1740de0ef55d6\": container with ID starting with 5f7166b9f2cffea47388c8a6a93e4cb2979ecf3e0663fa7fc6d1740de0ef55d6 not found: ID does not exist" containerID="5f7166b9f2cffea47388c8a6a93e4cb2979ecf3e0663fa7fc6d1740de0ef55d6" Oct 30 00:44:42.185094 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:42.185014 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f7166b9f2cffea47388c8a6a93e4cb2979ecf3e0663fa7fc6d1740de0ef55d6"} err="failed to get container status \"5f7166b9f2cffea47388c8a6a93e4cb2979ecf3e0663fa7fc6d1740de0ef55d6\": rpc error: code = NotFound desc = could not find container \"5f7166b9f2cffea47388c8a6a93e4cb2979ecf3e0663fa7fc6d1740de0ef55d6\": container with ID starting with 5f7166b9f2cffea47388c8a6a93e4cb2979ecf3e0663fa7fc6d1740de0ef55d6 not found: ID does not exist" Oct 30 00:44:42.230042 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:42.230012 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8698cc7897-jflvm"] Oct 30 00:44:42.243489 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:42.243450 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8698cc7897-jflvm"] Oct 30 00:44:42.307415 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:42.307385 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:42.323164 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:42.323138 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:43.197167 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:43.197138 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 00:44:43.817105 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:43.817071 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07e33ac0-f034-477b-8e5a-aaf82a23d7fa" path="/var/lib/kubelet/pods/07e33ac0-f034-477b-8e5a-aaf82a23d7fa/volumes" Oct 30 00:44:50.206047 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:50.206013 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:51.590585 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:44:51.590550 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:44:53.111341 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:53.111309 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:44:53.185749 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:44:53.185722 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:01.931555 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:01.931464 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-698568d77b-q4qgw"] Oct 30 00:45:01.931954 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:01.931750 5300 cpu_manager.go:414] "containerMap: removing container" podUID="07e33ac0-f034-477b-8e5a-aaf82a23d7fa" containerName="console" Oct 30 00:45:01.931954 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:01.931780 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07e33ac0-f034-477b-8e5a-aaf82a23d7fa" containerName="console" Oct 30 00:45:01.931954 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:01.931845 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="07e33ac0-f034-477b-8e5a-aaf82a23d7fa" containerName="console" Oct 30 00:45:01.934663 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:01.934643 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:01.954722 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:01.954683 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-698568d77b-q4qgw"] Oct 30 00:45:02.027104 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.027069 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:02.041668 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.041643 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09c85662-2f01-4d2d-9a04-24e3245549f3-oauth-serving-cert\") pod \"console-698568d77b-q4qgw\" (UID: \"09c85662-2f01-4d2d-9a04-24e3245549f3\") " pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:02.041741 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.041673 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09c85662-2f01-4d2d-9a04-24e3245549f3-console-config\") pod \"console-698568d77b-q4qgw\" (UID: \"09c85662-2f01-4d2d-9a04-24e3245549f3\") " pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:02.041741 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.041696 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09c85662-2f01-4d2d-9a04-24e3245549f3-console-serving-cert\") pod \"console-698568d77b-q4qgw\" (UID: \"09c85662-2f01-4d2d-9a04-24e3245549f3\") " pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:02.041845 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.041796 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09c85662-2f01-4d2d-9a04-24e3245549f3-trusted-ca-bundle\") pod \"console-698568d77b-q4qgw\" (UID: \"09c85662-2f01-4d2d-9a04-24e3245549f3\") " pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:02.041845 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.041833 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09c85662-2f01-4d2d-9a04-24e3245549f3-service-ca\") pod \"console-698568d77b-q4qgw\" (UID: \"09c85662-2f01-4d2d-9a04-24e3245549f3\") " pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:02.041924 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.041908 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9fl8\" (UniqueName: \"kubernetes.io/projected/09c85662-2f01-4d2d-9a04-24e3245549f3-kube-api-access-c9fl8\") pod \"console-698568d77b-q4qgw\" (UID: \"09c85662-2f01-4d2d-9a04-24e3245549f3\") " pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:02.041971 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.041943 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09c85662-2f01-4d2d-9a04-24e3245549f3-tmp\") pod \"console-698568d77b-q4qgw\" (UID: \"09c85662-2f01-4d2d-9a04-24e3245549f3\") " pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:02.042009 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.041972 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09c85662-2f01-4d2d-9a04-24e3245549f3-console-oauth-config\") pod \"console-698568d77b-q4qgw\" (UID: \"09c85662-2f01-4d2d-9a04-24e3245549f3\") " pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:02.127584 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.127540 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:02.142637 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.142608 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09c85662-2f01-4d2d-9a04-24e3245549f3-oauth-serving-cert\") pod \"console-698568d77b-q4qgw\" (UID: \"09c85662-2f01-4d2d-9a04-24e3245549f3\") " pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:02.142730 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.142642 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09c85662-2f01-4d2d-9a04-24e3245549f3-console-config\") pod \"console-698568d77b-q4qgw\" (UID: \"09c85662-2f01-4d2d-9a04-24e3245549f3\") " pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:02.142730 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.142660 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09c85662-2f01-4d2d-9a04-24e3245549f3-console-serving-cert\") pod \"console-698568d77b-q4qgw\" (UID: \"09c85662-2f01-4d2d-9a04-24e3245549f3\") " pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:02.142730 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.142677 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09c85662-2f01-4d2d-9a04-24e3245549f3-trusted-ca-bundle\") pod \"console-698568d77b-q4qgw\" (UID: \"09c85662-2f01-4d2d-9a04-24e3245549f3\") " pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:02.142730 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.142701 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09c85662-2f01-4d2d-9a04-24e3245549f3-service-ca\") pod \"console-698568d77b-q4qgw\" (UID: \"09c85662-2f01-4d2d-9a04-24e3245549f3\") " pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:02.142925 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.142742 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9fl8\" (UniqueName: \"kubernetes.io/projected/09c85662-2f01-4d2d-9a04-24e3245549f3-kube-api-access-c9fl8\") pod \"console-698568d77b-q4qgw\" (UID: \"09c85662-2f01-4d2d-9a04-24e3245549f3\") " pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:02.142925 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.142797 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09c85662-2f01-4d2d-9a04-24e3245549f3-tmp\") pod \"console-698568d77b-q4qgw\" (UID: \"09c85662-2f01-4d2d-9a04-24e3245549f3\") " pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:02.142925 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.142837 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09c85662-2f01-4d2d-9a04-24e3245549f3-console-oauth-config\") pod \"console-698568d77b-q4qgw\" (UID: \"09c85662-2f01-4d2d-9a04-24e3245549f3\") " pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:02.143295 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.143273 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09c85662-2f01-4d2d-9a04-24e3245549f3-tmp\") pod \"console-698568d77b-q4qgw\" (UID: \"09c85662-2f01-4d2d-9a04-24e3245549f3\") " pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:02.143529 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.143507 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09c85662-2f01-4d2d-9a04-24e3245549f3-console-config\") pod \"console-698568d77b-q4qgw\" (UID: \"09c85662-2f01-4d2d-9a04-24e3245549f3\") " pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:02.143529 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.143531 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09c85662-2f01-4d2d-9a04-24e3245549f3-service-ca\") pod \"console-698568d77b-q4qgw\" (UID: \"09c85662-2f01-4d2d-9a04-24e3245549f3\") " pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:02.143856 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.143829 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09c85662-2f01-4d2d-9a04-24e3245549f3-oauth-serving-cert\") pod \"console-698568d77b-q4qgw\" (UID: \"09c85662-2f01-4d2d-9a04-24e3245549f3\") " pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:02.143936 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.143914 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09c85662-2f01-4d2d-9a04-24e3245549f3-trusted-ca-bundle\") pod \"console-698568d77b-q4qgw\" (UID: \"09c85662-2f01-4d2d-9a04-24e3245549f3\") " pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:02.145403 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.145376 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09c85662-2f01-4d2d-9a04-24e3245549f3-console-oauth-config\") pod \"console-698568d77b-q4qgw\" (UID: \"09c85662-2f01-4d2d-9a04-24e3245549f3\") " pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:02.145403 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.145399 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09c85662-2f01-4d2d-9a04-24e3245549f3-console-serving-cert\") pod \"console-698568d77b-q4qgw\" (UID: \"09c85662-2f01-4d2d-9a04-24e3245549f3\") " pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:02.162599 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.162580 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9fl8\" (UniqueName: \"kubernetes.io/projected/09c85662-2f01-4d2d-9a04-24e3245549f3-kube-api-access-c9fl8\") pod \"console-698568d77b-q4qgw\" (UID: \"09c85662-2f01-4d2d-9a04-24e3245549f3\") " pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:02.245649 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.245618 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:02.312347 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.312301 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:02.386174 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.386139 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:02.535889 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:02.535818 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-698568d77b-q4qgw"] Oct 30 00:45:02.538565 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:45:02.538539 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c85662_2f01_4d2d_9a04_24e3245549f3.slice/crio-c65f650ec1adcb22ce870d38df159d2a947d55019815143633e64f7e48ffe4f8 WatchSource:0}: Error finding container c65f650ec1adcb22ce870d38df159d2a947d55019815143633e64f7e48ffe4f8: Status 404 returned error can't find the container with id c65f650ec1adcb22ce870d38df159d2a947d55019815143633e64f7e48ffe4f8 Oct 30 00:45:03.247409 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:03.247367 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-698568d77b-q4qgw" event={"ID":"09c85662-2f01-4d2d-9a04-24e3245549f3","Type":"ContainerStarted","Data":"f67310fe964169c7c879d8e3a692efef29d3cdb7fde3dd14291d76bf0a1a7752"} Oct 30 00:45:03.247859 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:03.247425 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-698568d77b-q4qgw" event={"ID":"09c85662-2f01-4d2d-9a04-24e3245549f3","Type":"ContainerStarted","Data":"c65f650ec1adcb22ce870d38df159d2a947d55019815143633e64f7e48ffe4f8"} Oct 30 00:45:03.288348 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:03.288290 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-698568d77b-q4qgw" podStartSLOduration=2.288274561 podStartE2EDuration="2.288274561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:03.288014491 +0000 UTC m=+103.603882817" watchObservedRunningTime="2026-10-30 00:45:03.288274561 +0000 UTC m=+103.604142885" Oct 30 00:45:05.609159 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:05.609118 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:05.684226 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:05.684197 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:12.245837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:12.245800 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:12.246368 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:12.245867 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:12.252078 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:12.252053 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:12.279484 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:12.279458 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-698568d77b-q4qgw" Oct 30 00:45:12.359689 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:12.359660 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76d5554bbf-5dg2c"] Oct 30 00:45:15.712911 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:15.712881 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:15.813255 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:15.813217 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:15.884352 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:15.884325 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:20.869909 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:20.869865 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:20.948990 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:20.948959 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:21.593238 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:45:21.593203 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:45:28.366337 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:28.366304 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:37.379072 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:37.379016 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-76d5554bbf-5dg2c" podUID="202bf6a4-7b22-4988-893f-ca280f78fca5" containerName="console" containerID="cri-o://cd26ea187c8957717b7c7cf28807617fb0dbeaab274f0cab1aa034dc85c1f145" gracePeriod=15 Oct 30 00:45:37.608088 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:37.608065 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:45:37.713495 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:37.713416 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-console-config\" (UniqueName: \"kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-console-config\") pod \"202bf6a4-7b22-4988-893f-ca280f78fca5\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " Oct 30 00:45:37.713495 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:37.713467 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/202bf6a4-7b22-4988-893f-ca280f78fca5-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/202bf6a4-7b22-4988-893f-ca280f78fca5-console-oauth-config\") pod \"202bf6a4-7b22-4988-893f-ca280f78fca5\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " Oct 30 00:45:37.713701 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:37.713517 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-service-ca\" (UniqueName: \"kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-service-ca\") pod \"202bf6a4-7b22-4988-893f-ca280f78fca5\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " Oct 30 00:45:37.713701 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:37.713543 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-trusted-ca-bundle\") pod \"202bf6a4-7b22-4988-893f-ca280f78fca5\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " Oct 30 00:45:37.713701 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:37.713586 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-oauth-serving-cert\") pod \"202bf6a4-7b22-4988-893f-ca280f78fca5\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " Oct 30 00:45:37.713701 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:37.713614 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/202bf6a4-7b22-4988-893f-ca280f78fca5-kube-api-access-fgvr7\" (UniqueName: \"kubernetes.io/projected/202bf6a4-7b22-4988-893f-ca280f78fca5-kube-api-access-fgvr7\") pod \"202bf6a4-7b22-4988-893f-ca280f78fca5\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " Oct 30 00:45:37.713701 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:37.713684 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/202bf6a4-7b22-4988-893f-ca280f78fca5-tmp\" (UniqueName: \"kubernetes.io/empty-dir/202bf6a4-7b22-4988-893f-ca280f78fca5-tmp\") pod \"202bf6a4-7b22-4988-893f-ca280f78fca5\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " Oct 30 00:45:37.713974 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:37.713722 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/202bf6a4-7b22-4988-893f-ca280f78fca5-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/202bf6a4-7b22-4988-893f-ca280f78fca5-console-serving-cert\") pod \"202bf6a4-7b22-4988-893f-ca280f78fca5\" (UID: \"202bf6a4-7b22-4988-893f-ca280f78fca5\") " Oct 30 00:45:37.714033 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:37.713864 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-console-config" pod "202bf6a4-7b22-4988-893f-ca280f78fca5" (UID: "202bf6a4-7b22-4988-893f-ca280f78fca5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:45:37.714144 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:37.714048 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202bf6a4-7b22-4988-893f-ca280f78fca5-tmp" pod "202bf6a4-7b22-4988-893f-ca280f78fca5" (UID: "202bf6a4-7b22-4988-893f-ca280f78fca5"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:45:37.714144 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:37.714101 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-service-ca" pod "202bf6a4-7b22-4988-893f-ca280f78fca5" (UID: "202bf6a4-7b22-4988-893f-ca280f78fca5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:45:37.714144 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:37.714122 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-oauth-serving-cert" pod "202bf6a4-7b22-4988-893f-ca280f78fca5" (UID: "202bf6a4-7b22-4988-893f-ca280f78fca5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:45:37.714276 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:37.714202 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-trusted-ca-bundle" pod "202bf6a4-7b22-4988-893f-ca280f78fca5" (UID: "202bf6a4-7b22-4988-893f-ca280f78fca5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:45:37.715865 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:37.715839 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/202bf6a4-7b22-4988-893f-ca280f78fca5-console-serving-cert" pod "202bf6a4-7b22-4988-893f-ca280f78fca5" (UID: "202bf6a4-7b22-4988-893f-ca280f78fca5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:45:37.716236 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:37.716207 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/202bf6a4-7b22-4988-893f-ca280f78fca5-console-oauth-config" pod "202bf6a4-7b22-4988-893f-ca280f78fca5" (UID: "202bf6a4-7b22-4988-893f-ca280f78fca5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:45:37.716236 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:37.716211 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202bf6a4-7b22-4988-893f-ca280f78fca5-kube-api-access-fgvr7" pod "202bf6a4-7b22-4988-893f-ca280f78fca5" (UID: "202bf6a4-7b22-4988-893f-ca280f78fca5"). InnerVolumeSpecName "kube-api-access-fgvr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:45:37.814649 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:37.814623 5300 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:45:37.814649 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:37.814645 5300 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-trusted-ca-bundle\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:45:37.814649 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:37.814655 5300 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-oauth-serving-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:45:37.814864 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:37.814664 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgvr7\" (UniqueName: \"kubernetes.io/projected/202bf6a4-7b22-4988-893f-ca280f78fca5-kube-api-access-fgvr7\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:45:37.814864 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:37.814675 5300 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/202bf6a4-7b22-4988-893f-ca280f78fca5-tmp\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:45:37.814864 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:37.814684 5300 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/202bf6a4-7b22-4988-893f-ca280f78fca5-console-serving-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:45:37.814864 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:37.814693 5300 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/202bf6a4-7b22-4988-893f-ca280f78fca5-console-config\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:45:37.814864 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:37.814703 5300 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/202bf6a4-7b22-4988-893f-ca280f78fca5-console-oauth-config\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:45:38.364299 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:38.364264 5300 generic.go:356] "Generic (PLEG): container finished" podID="202bf6a4-7b22-4988-893f-ca280f78fca5" containerID="cd26ea187c8957717b7c7cf28807617fb0dbeaab274f0cab1aa034dc85c1f145" exitCode=0 Oct 30 00:45:38.364467 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:38.364316 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76d5554bbf-5dg2c" Oct 30 00:45:38.364467 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:38.364356 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76d5554bbf-5dg2c" event={"ID":"202bf6a4-7b22-4988-893f-ca280f78fca5","Type":"ContainerDied","Data":"cd26ea187c8957717b7c7cf28807617fb0dbeaab274f0cab1aa034dc85c1f145"} Oct 30 00:45:38.364467 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:38.364415 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76d5554bbf-5dg2c" event={"ID":"202bf6a4-7b22-4988-893f-ca280f78fca5","Type":"ContainerDied","Data":"1fedba3b88ff500a1f96803731a39ea057a0e8f3f3a0c30dea8a31436bade1c2"} Oct 30 00:45:38.364467 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:38.364440 5300 scope.go:122] "RemoveContainer" containerID="cd26ea187c8957717b7c7cf28807617fb0dbeaab274f0cab1aa034dc85c1f145" Oct 30 00:45:38.371968 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:38.371954 5300 scope.go:122] "RemoveContainer" containerID="cd26ea187c8957717b7c7cf28807617fb0dbeaab274f0cab1aa034dc85c1f145" Oct 30 00:45:38.372210 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:45:38.372194 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd26ea187c8957717b7c7cf28807617fb0dbeaab274f0cab1aa034dc85c1f145\": container with ID starting with cd26ea187c8957717b7c7cf28807617fb0dbeaab274f0cab1aa034dc85c1f145 not found: ID does not exist" containerID="cd26ea187c8957717b7c7cf28807617fb0dbeaab274f0cab1aa034dc85c1f145" Oct 30 00:45:38.372262 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:38.372219 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd26ea187c8957717b7c7cf28807617fb0dbeaab274f0cab1aa034dc85c1f145"} err="failed to get container status \"cd26ea187c8957717b7c7cf28807617fb0dbeaab274f0cab1aa034dc85c1f145\": rpc error: code = NotFound desc = could not find container \"cd26ea187c8957717b7c7cf28807617fb0dbeaab274f0cab1aa034dc85c1f145\": container with ID starting with cd26ea187c8957717b7c7cf28807617fb0dbeaab274f0cab1aa034dc85c1f145 not found: ID does not exist" Oct 30 00:45:38.416928 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:38.416903 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76d5554bbf-5dg2c"] Oct 30 00:45:38.430599 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:38.430575 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76d5554bbf-5dg2c"] Oct 30 00:45:39.816757 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:39.816727 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="202bf6a4-7b22-4988-893f-ca280f78fca5" path="/var/lib/kubelet/pods/202bf6a4-7b22-4988-893f-ca280f78fca5/volumes" Oct 30 00:45:51.589679 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:45:51.589645 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:45:53.690794 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:53.690747 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:45:53.768861 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:45:53.768832 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:06.095049 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:46:06.094965 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:21.591191 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:46:21.591149 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:46:28.520006 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:46:28.519972 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:47.157447 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:46:47.157411 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:47.258549 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:46:47.258517 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:46:51.590476 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:46:51.590436 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:47:08.612917 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:08.612888 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:09.972040 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:09.972006 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:11.817521 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:11.817494 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:11.919752 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:11.919692 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:14.358412 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:14.358384 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:14.458380 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:14.458345 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:14.733568 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:14.733531 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:14.862575 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:14.862550 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:16.716892 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:16.716863 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:16.951857 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:16.951824 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:16.977628 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:16.977607 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-4z84c"] Oct 30 00:47:16.977874 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:16.977862 5300 cpu_manager.go:414] "containerMap: removing container" podUID="202bf6a4-7b22-4988-893f-ca280f78fca5" containerName="console" Oct 30 00:47:16.977919 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:16.977875 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="202bf6a4-7b22-4988-893f-ca280f78fca5" containerName="console" Oct 30 00:47:16.977951 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:16.977934 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="202bf6a4-7b22-4988-893f-ca280f78fca5" containerName="console" Oct 30 00:47:16.980551 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:16.980537 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4z84c" Oct 30 00:47:16.998627 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:16.998604 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-4z84c"] Oct 30 00:47:17.004776 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:17.004742 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js57w\" (UniqueName: \"kubernetes.io/projected/3b20ffb8-1a6c-40a8-b0b7-c74fdc0ca3d9-kube-api-access-js57w\") pod \"cert-manager-cainjector-5c57bbd4fc-4z84c\" (UID: \"3b20ffb8-1a6c-40a8-b0b7-c74fdc0ca3d9\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4z84c" Oct 30 00:47:17.004873 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:17.004840 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b20ffb8-1a6c-40a8-b0b7-c74fdc0ca3d9-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-4z84c\" (UID: \"3b20ffb8-1a6c-40a8-b0b7-c74fdc0ca3d9\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4z84c" Oct 30 00:47:17.012811 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:17.012791 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 00:47:17.012885 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:17.012848 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 00:47:17.012929 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:17.012892 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-mzz6m\"" Oct 30 00:47:17.105378 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:17.105348 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b20ffb8-1a6c-40a8-b0b7-c74fdc0ca3d9-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-4z84c\" (UID: \"3b20ffb8-1a6c-40a8-b0b7-c74fdc0ca3d9\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4z84c" Oct 30 00:47:17.105521 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:17.105404 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js57w\" (UniqueName: \"kubernetes.io/projected/3b20ffb8-1a6c-40a8-b0b7-c74fdc0ca3d9-kube-api-access-js57w\") pod \"cert-manager-cainjector-5c57bbd4fc-4z84c\" (UID: \"3b20ffb8-1a6c-40a8-b0b7-c74fdc0ca3d9\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4z84c" Oct 30 00:47:17.125787 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:17.125734 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b20ffb8-1a6c-40a8-b0b7-c74fdc0ca3d9-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-4z84c\" (UID: \"3b20ffb8-1a6c-40a8-b0b7-c74fdc0ca3d9\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4z84c" Oct 30 00:47:17.125907 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:17.125843 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js57w\" (UniqueName: \"kubernetes.io/projected/3b20ffb8-1a6c-40a8-b0b7-c74fdc0ca3d9-kube-api-access-js57w\") pod \"cert-manager-cainjector-5c57bbd4fc-4z84c\" (UID: \"3b20ffb8-1a6c-40a8-b0b7-c74fdc0ca3d9\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4z84c" Oct 30 00:47:17.301227 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:17.301144 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4z84c" Oct 30 00:47:17.590198 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:17.590113 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-4z84c"] Oct 30 00:47:17.593426 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:47:17.593401 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b20ffb8_1a6c_40a8_b0b7_c74fdc0ca3d9.slice/crio-f15777e48879ae690e798139eb7b31858337bf955086c4764ae2e27cd51689b1 WatchSource:0}: Error finding container f15777e48879ae690e798139eb7b31858337bf955086c4764ae2e27cd51689b1: Status 404 returned error can't find the container with id f15777e48879ae690e798139eb7b31858337bf955086c4764ae2e27cd51689b1 Oct 30 00:47:17.684682 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:17.684655 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4z84c" event={"ID":"3b20ffb8-1a6c-40a8-b0b7-c74fdc0ca3d9","Type":"ContainerStarted","Data":"f15777e48879ae690e798139eb7b31858337bf955086c4764ae2e27cd51689b1"} Oct 30 00:47:18.804954 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:18.804920 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:18.905334 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:18.905299 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:21.590460 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:47:21.590419 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:47:21.700549 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:21.700514 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4z84c" event={"ID":"3b20ffb8-1a6c-40a8-b0b7-c74fdc0ca3d9","Type":"ContainerStarted","Data":"4a4d41d3f4d396010eea128b4780d2d2ffa9b2ddb3cf586c5dfa803c3c349eb2"} Oct 30 00:47:21.740262 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:21.740210 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4z84c" podStartSLOduration=2.44569451 podStartE2EDuration="5.74019803s" totalImagesPullingTime="3.29450352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:21.739435678 +0000 UTC m=+242.055304012" watchObservedRunningTime="2026-10-30 00:47:21.74019803 +0000 UTC m=+242.056066351" Oct 30 00:47:22.087700 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:22.087662 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:22.188049 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:22.188016 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:23.942256 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:23.942220 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:26.217317 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:26.217281 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:26.306985 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:26.306948 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:28.595627 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:28.595594 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:32.268829 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:32.268794 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:32.368696 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:32.368667 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:34.409270 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:34.409238 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:48.811365 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:48.811333 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:48.967412 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:48.967352 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:48.993701 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:48.993668 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-85798cdd6b-lh96s"] Oct 30 00:47:49.000058 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.000037 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-lh96s" Oct 30 00:47:49.018922 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.018899 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-85798cdd6b-lh96s"] Oct 30 00:47:49.031359 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.031332 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 30 00:47:49.031359 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.031335 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 30 00:47:49.031359 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.031335 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Oct 30 00:47:49.031528 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.031370 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Oct 30 00:47:49.031528 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.031432 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Oct 30 00:47:49.031588 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.031573 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-srkcr\"" Oct 30 00:47:49.042416 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.042392 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/072ce660-81f0-44ed-b3ee-917e3bfc687e-manager-config\") pod \"lws-controller-manager-85798cdd6b-lh96s\" (UID: \"072ce660-81f0-44ed-b3ee-917e3bfc687e\") " pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-lh96s" Oct 30 00:47:49.042504 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.042422 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/072ce660-81f0-44ed-b3ee-917e3bfc687e-metrics-cert\") pod \"lws-controller-manager-85798cdd6b-lh96s\" (UID: \"072ce660-81f0-44ed-b3ee-917e3bfc687e\") " pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-lh96s" Oct 30 00:47:49.042504 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.042443 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzzgl\" (UniqueName: \"kubernetes.io/projected/072ce660-81f0-44ed-b3ee-917e3bfc687e-kube-api-access-fzzgl\") pod \"lws-controller-manager-85798cdd6b-lh96s\" (UID: \"072ce660-81f0-44ed-b3ee-917e3bfc687e\") " pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-lh96s" Oct 30 00:47:49.042570 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.042510 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/072ce660-81f0-44ed-b3ee-917e3bfc687e-cert\") pod \"lws-controller-manager-85798cdd6b-lh96s\" (UID: \"072ce660-81f0-44ed-b3ee-917e3bfc687e\") " pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-lh96s" Oct 30 00:47:49.088042 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.087964 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:49.143271 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.143240 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/072ce660-81f0-44ed-b3ee-917e3bfc687e-manager-config\") pod \"lws-controller-manager-85798cdd6b-lh96s\" (UID: \"072ce660-81f0-44ed-b3ee-917e3bfc687e\") " pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-lh96s" Oct 30 00:47:49.143271 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.143274 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/072ce660-81f0-44ed-b3ee-917e3bfc687e-metrics-cert\") pod \"lws-controller-manager-85798cdd6b-lh96s\" (UID: \"072ce660-81f0-44ed-b3ee-917e3bfc687e\") " pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-lh96s" Oct 30 00:47:49.143492 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.143295 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzzgl\" (UniqueName: \"kubernetes.io/projected/072ce660-81f0-44ed-b3ee-917e3bfc687e-kube-api-access-fzzgl\") pod \"lws-controller-manager-85798cdd6b-lh96s\" (UID: \"072ce660-81f0-44ed-b3ee-917e3bfc687e\") " pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-lh96s" Oct 30 00:47:49.143492 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.143321 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/072ce660-81f0-44ed-b3ee-917e3bfc687e-cert\") pod \"lws-controller-manager-85798cdd6b-lh96s\" (UID: \"072ce660-81f0-44ed-b3ee-917e3bfc687e\") " pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-lh96s" Oct 30 00:47:49.144081 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.144054 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/072ce660-81f0-44ed-b3ee-917e3bfc687e-manager-config\") pod \"lws-controller-manager-85798cdd6b-lh96s\" (UID: \"072ce660-81f0-44ed-b3ee-917e3bfc687e\") " pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-lh96s" Oct 30 00:47:49.145866 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.145844 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/072ce660-81f0-44ed-b3ee-917e3bfc687e-metrics-cert\") pod \"lws-controller-manager-85798cdd6b-lh96s\" (UID: \"072ce660-81f0-44ed-b3ee-917e3bfc687e\") " pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-lh96s" Oct 30 00:47:49.146002 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.145982 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/072ce660-81f0-44ed-b3ee-917e3bfc687e-cert\") pod \"lws-controller-manager-85798cdd6b-lh96s\" (UID: \"072ce660-81f0-44ed-b3ee-917e3bfc687e\") " pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-lh96s" Oct 30 00:47:49.165005 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.164975 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzzgl\" (UniqueName: \"kubernetes.io/projected/072ce660-81f0-44ed-b3ee-917e3bfc687e-kube-api-access-fzzgl\") pod \"lws-controller-manager-85798cdd6b-lh96s\" (UID: \"072ce660-81f0-44ed-b3ee-917e3bfc687e\") " pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-lh96s" Oct 30 00:47:49.311027 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.310994 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-lh96s" Oct 30 00:47:49.603559 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.603528 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-85798cdd6b-lh96s"] Oct 30 00:47:49.607907 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:47:49.607843 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod072ce660_81f0_44ed_b3ee_917e3bfc687e.slice/crio-810364dd4be1af2ccea7d3ad381467a066360469d078957ac34052b752e6949a WatchSource:0}: Error finding container 810364dd4be1af2ccea7d3ad381467a066360469d078957ac34052b752e6949a: Status 404 returned error can't find the container with id 810364dd4be1af2ccea7d3ad381467a066360469d078957ac34052b752e6949a Oct 30 00:47:49.612194 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.612169 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:49.791423 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.791393 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-lh96s" event={"ID":"072ce660-81f0-44ed-b3ee-917e3bfc687e","Type":"ContainerStarted","Data":"810364dd4be1af2ccea7d3ad381467a066360469d078957ac34052b752e6949a"} Oct 30 00:47:49.796521 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.796496 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58"] Oct 30 00:47:49.800828 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.800812 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58" Oct 30 00:47:49.837506 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.837484 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58"] Oct 30 00:47:49.849037 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.849016 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f5fb34d6-45e8-4caa-95ef-6f801d88a584-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84dd548884-rfc58\" (UID: \"f5fb34d6-45e8-4caa-95ef-6f801d88a584\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58" Oct 30 00:47:49.849145 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.849051 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5fb34d6-45e8-4caa-95ef-6f801d88a584-tmp\") pod \"opendatahub-operator-controller-manager-84dd548884-rfc58\" (UID: \"f5fb34d6-45e8-4caa-95ef-6f801d88a584\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58" Oct 30 00:47:49.849145 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.849078 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f5fb34d6-45e8-4caa-95ef-6f801d88a584-metrics-cert\") pod \"opendatahub-operator-controller-manager-84dd548884-rfc58\" (UID: \"f5fb34d6-45e8-4caa-95ef-6f801d88a584\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58" Oct 30 00:47:49.849220 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.849139 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f5fb34d6-45e8-4caa-95ef-6f801d88a584-manifests\") pod \"opendatahub-operator-controller-manager-84dd548884-rfc58\" (UID: \"f5fb34d6-45e8-4caa-95ef-6f801d88a584\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58" Oct 30 00:47:49.849277 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.849264 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f5fb34d6-45e8-4caa-95ef-6f801d88a584-webhook-cert\") pod \"opendatahub-operator-controller-manager-84dd548884-rfc58\" (UID: \"f5fb34d6-45e8-4caa-95ef-6f801d88a584\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58" Oct 30 00:47:49.849311 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.849300 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf7sp\" (UniqueName: \"kubernetes.io/projected/f5fb34d6-45e8-4caa-95ef-6f801d88a584-kube-api-access-zf7sp\") pod \"opendatahub-operator-controller-manager-84dd548884-rfc58\" (UID: \"f5fb34d6-45e8-4caa-95ef-6f801d88a584\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58" Oct 30 00:47:49.895034 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.894964 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 00:47:49.895034 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.895010 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 00:47:49.895034 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.895011 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 00:47:49.895216 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.895118 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-h7rmk\"" Oct 30 00:47:49.895216 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.895161 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 00:47:49.895216 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.895203 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 00:47:49.913064 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.913043 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:49.950207 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.950182 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f5fb34d6-45e8-4caa-95ef-6f801d88a584-webhook-cert\") pod \"opendatahub-operator-controller-manager-84dd548884-rfc58\" (UID: \"f5fb34d6-45e8-4caa-95ef-6f801d88a584\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58" Oct 30 00:47:49.950295 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.950219 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf7sp\" (UniqueName: \"kubernetes.io/projected/f5fb34d6-45e8-4caa-95ef-6f801d88a584-kube-api-access-zf7sp\") pod \"opendatahub-operator-controller-manager-84dd548884-rfc58\" (UID: \"f5fb34d6-45e8-4caa-95ef-6f801d88a584\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58" Oct 30 00:47:49.950295 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.950237 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f5fb34d6-45e8-4caa-95ef-6f801d88a584-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84dd548884-rfc58\" (UID: \"f5fb34d6-45e8-4caa-95ef-6f801d88a584\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58" Oct 30 00:47:49.950295 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.950277 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5fb34d6-45e8-4caa-95ef-6f801d88a584-tmp\") pod \"opendatahub-operator-controller-manager-84dd548884-rfc58\" (UID: \"f5fb34d6-45e8-4caa-95ef-6f801d88a584\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58" Oct 30 00:47:49.950437 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.950312 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f5fb34d6-45e8-4caa-95ef-6f801d88a584-metrics-cert\") pod \"opendatahub-operator-controller-manager-84dd548884-rfc58\" (UID: \"f5fb34d6-45e8-4caa-95ef-6f801d88a584\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58" Oct 30 00:47:49.950437 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.950335 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f5fb34d6-45e8-4caa-95ef-6f801d88a584-manifests\") pod \"opendatahub-operator-controller-manager-84dd548884-rfc58\" (UID: \"f5fb34d6-45e8-4caa-95ef-6f801d88a584\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58" Oct 30 00:47:49.950714 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.950679 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f5fb34d6-45e8-4caa-95ef-6f801d88a584-manifests\") pod \"opendatahub-operator-controller-manager-84dd548884-rfc58\" (UID: \"f5fb34d6-45e8-4caa-95ef-6f801d88a584\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58" Oct 30 00:47:49.952645 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.952624 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f5fb34d6-45e8-4caa-95ef-6f801d88a584-metrics-cert\") pod \"opendatahub-operator-controller-manager-84dd548884-rfc58\" (UID: \"f5fb34d6-45e8-4caa-95ef-6f801d88a584\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58" Oct 30 00:47:49.952727 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.952678 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f5fb34d6-45e8-4caa-95ef-6f801d88a584-webhook-cert\") pod \"opendatahub-operator-controller-manager-84dd548884-rfc58\" (UID: \"f5fb34d6-45e8-4caa-95ef-6f801d88a584\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58" Oct 30 00:47:49.952876 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.952860 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5fb34d6-45e8-4caa-95ef-6f801d88a584-tmp\") pod \"opendatahub-operator-controller-manager-84dd548884-rfc58\" (UID: \"f5fb34d6-45e8-4caa-95ef-6f801d88a584\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58" Oct 30 00:47:49.953058 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.953042 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f5fb34d6-45e8-4caa-95ef-6f801d88a584-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84dd548884-rfc58\" (UID: \"f5fb34d6-45e8-4caa-95ef-6f801d88a584\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58" Oct 30 00:47:49.979801 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:49.979780 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf7sp\" (UniqueName: \"kubernetes.io/projected/f5fb34d6-45e8-4caa-95ef-6f801d88a584-kube-api-access-zf7sp\") pod \"opendatahub-operator-controller-manager-84dd548884-rfc58\" (UID: \"f5fb34d6-45e8-4caa-95ef-6f801d88a584\") " pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58" Oct 30 00:47:50.037241 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:50.037212 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:50.115994 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:50.115959 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58" Oct 30 00:47:50.135686 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:50.135659 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:50.235489 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:50.235464 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:50.459247 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:47:50.459210 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5fb34d6_45e8_4caa_95ef_6f801d88a584.slice/crio-2939f24657d75ded20efc4fe3307aca3fdd6b88104d43f421ed811727980c688 WatchSource:0}: Error finding container 2939f24657d75ded20efc4fe3307aca3fdd6b88104d43f421ed811727980c688: Status 404 returned error can't find the container with id 2939f24657d75ded20efc4fe3307aca3fdd6b88104d43f421ed811727980c688 Oct 30 00:47:50.469486 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:50.469460 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58"] Oct 30 00:47:50.795520 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:50.795489 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58" event={"ID":"f5fb34d6-45e8-4caa-95ef-6f801d88a584","Type":"ContainerStarted","Data":"2939f24657d75ded20efc4fe3307aca3fdd6b88104d43f421ed811727980c688"} Oct 30 00:47:51.096122 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:51.096038 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:47:51.597368 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:47:51.597329 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:47:53.817679 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:53.817634 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58" event={"ID":"f5fb34d6-45e8-4caa-95ef-6f801d88a584","Type":"ContainerStarted","Data":"831d5055663f3ed23ed9f9d60b7fbc34272d692b575824980ba364c236bcead4"} Oct 30 00:47:54.817626 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:54.817589 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-lh96s" event={"ID":"072ce660-81f0-44ed-b3ee-917e3bfc687e","Type":"ContainerStarted","Data":"6357a70a8e57baf2b28a1ee27fad01b96f739cc976fff198dda5400fee17971d"} Oct 30 00:47:54.817852 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:54.817724 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-lh96s" Oct 30 00:47:54.819311 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:54.819287 5300 generic.go:356] "Generic (PLEG): container finished" podID="f5fb34d6-45e8-4caa-95ef-6f801d88a584" containerID="831d5055663f3ed23ed9f9d60b7fbc34272d692b575824980ba364c236bcead4" exitCode=0 Oct 30 00:47:54.819423 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:54.819322 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58" event={"ID":"f5fb34d6-45e8-4caa-95ef-6f801d88a584","Type":"ContainerDied","Data":"831d5055663f3ed23ed9f9d60b7fbc34272d692b575824980ba364c236bcead4"} Oct 30 00:47:54.857124 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:54.857076 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-lh96s" podStartSLOduration=2.549953055 podStartE2EDuration="6.857063363s" totalImagesPullingTime="4.307110308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:54.856577156 +0000 UTC m=+275.172445480" watchObservedRunningTime="2026-10-30 00:47:54.857063363 +0000 UTC m=+275.172931750" Oct 30 00:47:55.824139 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:55.824102 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58" event={"ID":"f5fb34d6-45e8-4caa-95ef-6f801d88a584","Type":"ContainerStarted","Data":"253942264e62f4042c887227130be0dc794c59b994eb4140db5e338479f397cd"} Oct 30 00:47:55.902746 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:55.902698 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58" podStartSLOduration=3.119826709 podStartE2EDuration="6.90268561s" totalImagesPullingTime="3.266388657s" totalInitContainerRuntime="516.470244ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:55.888880486 +0000 UTC m=+276.204748808" watchObservedRunningTime="2026-10-30 00:47:55.90268561 +0000 UTC m=+276.218553934" Oct 30 00:47:56.827349 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:47:56.827322 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58" Oct 30 00:48:05.826214 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:05.826142 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-85798cdd6b-lh96s" Oct 30 00:48:07.835248 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:07.835225 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-84dd548884-rfc58" Oct 30 00:48:10.227574 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:10.227534 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:10.325578 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:10.325540 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:10.789664 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:10.789629 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:10.800462 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:10.800437 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6cfd9d8567-zfw8x"] Oct 30 00:48:10.803515 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:10.803500 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-zfw8x" Oct 30 00:48:10.819133 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:10.819102 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 00:48:10.832273 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:10.832254 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 00:48:10.833477 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:10.833450 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6cfd9d8567-zfw8x"] Oct 30 00:48:10.911350 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:10.911301 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 00:48:10.911350 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:10.911338 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 00:48:10.911563 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:10.911391 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7qwfp\"" Oct 30 00:48:10.930722 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:10.930700 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c45710be-fadb-4638-a39a-c84664585e3c-tls-certs\") pod \"kube-auth-proxy-6cfd9d8567-zfw8x\" (UID: \"c45710be-fadb-4638-a39a-c84664585e3c\") " pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-zfw8x" Oct 30 00:48:10.930797 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:10.930747 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wtms\" (UniqueName: \"kubernetes.io/projected/c45710be-fadb-4638-a39a-c84664585e3c-kube-api-access-6wtms\") pod \"kube-auth-proxy-6cfd9d8567-zfw8x\" (UID: \"c45710be-fadb-4638-a39a-c84664585e3c\") " pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-zfw8x" Oct 30 00:48:10.930797 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:10.930790 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c45710be-fadb-4638-a39a-c84664585e3c-tmp\") pod \"kube-auth-proxy-6cfd9d8567-zfw8x\" (UID: \"c45710be-fadb-4638-a39a-c84664585e3c\") " pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-zfw8x" Oct 30 00:48:11.032086 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:11.032039 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wtms\" (UniqueName: \"kubernetes.io/projected/c45710be-fadb-4638-a39a-c84664585e3c-kube-api-access-6wtms\") pod \"kube-auth-proxy-6cfd9d8567-zfw8x\" (UID: \"c45710be-fadb-4638-a39a-c84664585e3c\") " pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-zfw8x" Oct 30 00:48:11.032086 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:11.032090 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c45710be-fadb-4638-a39a-c84664585e3c-tmp\") pod \"kube-auth-proxy-6cfd9d8567-zfw8x\" (UID: \"c45710be-fadb-4638-a39a-c84664585e3c\") " pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-zfw8x" Oct 30 00:48:11.032307 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:11.032158 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c45710be-fadb-4638-a39a-c84664585e3c-tls-certs\") pod \"kube-auth-proxy-6cfd9d8567-zfw8x\" (UID: \"c45710be-fadb-4638-a39a-c84664585e3c\") " pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-zfw8x" Oct 30 00:48:11.032367 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:48:11.032305 5300 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Oct 30 00:48:11.032419 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:48:11.032409 5300 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c45710be-fadb-4638-a39a-c84664585e3c-tls-certs podName:c45710be-fadb-4638-a39a-c84664585e3c nodeName:}" failed. No retries permitted until 2026-10-30 00:48:11.532374268 +0000 UTC m=+291.848242592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/c45710be-fadb-4638-a39a-c84664585e3c-tls-certs") pod "kube-auth-proxy-6cfd9d8567-zfw8x" (UID: "c45710be-fadb-4638-a39a-c84664585e3c") : secret "kube-auth-proxy-tls" not found Oct 30 00:48:11.034322 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:11.034300 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c45710be-fadb-4638-a39a-c84664585e3c-tmp\") pod \"kube-auth-proxy-6cfd9d8567-zfw8x\" (UID: \"c45710be-fadb-4638-a39a-c84664585e3c\") " pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-zfw8x" Oct 30 00:48:11.054945 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:11.054885 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wtms\" (UniqueName: \"kubernetes.io/projected/c45710be-fadb-4638-a39a-c84664585e3c-kube-api-access-6wtms\") pod \"kube-auth-proxy-6cfd9d8567-zfw8x\" (UID: \"c45710be-fadb-4638-a39a-c84664585e3c\") " pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-zfw8x" Oct 30 00:48:11.232221 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:11.232185 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:11.536365 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:11.536324 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c45710be-fadb-4638-a39a-c84664585e3c-tls-certs\") pod \"kube-auth-proxy-6cfd9d8567-zfw8x\" (UID: \"c45710be-fadb-4638-a39a-c84664585e3c\") " pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-zfw8x" Oct 30 00:48:11.538734 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:11.538709 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c45710be-fadb-4638-a39a-c84664585e3c-tls-certs\") pod \"kube-auth-proxy-6cfd9d8567-zfw8x\" (UID: \"c45710be-fadb-4638-a39a-c84664585e3c\") " pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-zfw8x" Oct 30 00:48:11.716206 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:11.716172 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-zfw8x" Oct 30 00:48:12.010503 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:12.010475 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6cfd9d8567-zfw8x"] Oct 30 00:48:12.013650 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:48:12.013617 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45710be_fadb_4638_a39a_c84664585e3c.slice/crio-f7f7236acec488932405a976a3bff6ae176d23a5973010ee5050825efb392020 WatchSource:0}: Error finding container f7f7236acec488932405a976a3bff6ae176d23a5973010ee5050825efb392020: Status 404 returned error can't find the container with id f7f7236acec488932405a976a3bff6ae176d23a5973010ee5050825efb392020 Oct 30 00:48:12.050001 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:12.049977 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:12.147383 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:12.147350 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:12.880243 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:12.880203 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-zfw8x" event={"ID":"c45710be-fadb-4638-a39a-c84664585e3c","Type":"ContainerStarted","Data":"f7f7236acec488932405a976a3bff6ae176d23a5973010ee5050825efb392020"} Oct 30 00:48:12.958170 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:12.958139 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:14.053020 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:14.052977 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:14.438061 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:14.437980 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:14.703692 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:14.703610 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:14.888492 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:14.888457 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-zfw8x" event={"ID":"c45710be-fadb-4638-a39a-c84664585e3c","Type":"ContainerStarted","Data":"31dfb81bf25d4ec81156293a4ea73ae9a55c6987832500ee0e41e8cfe4a664db"} Oct 30 00:48:14.928274 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:14.928225 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6cfd9d8567-zfw8x" podStartSLOduration=2.693125133 podStartE2EDuration="4.92821152s" totalImagesPullingTime="2.235086387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:14.927307532 +0000 UTC m=+295.243175857" watchObservedRunningTime="2026-10-30 00:48:14.92821152 +0000 UTC m=+295.244079844" Oct 30 00:48:15.885042 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:15.885008 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:15.984841 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:15.984809 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:16.254103 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:16.254073 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:16.557703 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:16.557603 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:16.688859 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:16.688814 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:21.590880 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:48:21.590843 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:48:27.724226 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:27.724185 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:27.882214 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:27.882181 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:28.728946 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:28.728902 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:46.078856 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:46.078820 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:49.030933 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:48:49.030904 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:48:51.590581 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:48:51.590546 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:49:09.273489 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:49:09.273451 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:09.373391 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:49:09.373360 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:10.112867 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:49:10.112832 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:14.316316 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:49:14.316285 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:14.415413 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:49:14.415381 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:14.517070 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:49:14.517041 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:14.619354 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:49:14.619282 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:14.733097 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:49:14.733069 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:14.896653 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:49:14.896574 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:15.032165 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:49:15.032130 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:15.157606 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:49:15.157531 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:17.332343 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:49:17.332306 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:17.432229 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:49:17.432194 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:17.532605 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:49:17.532576 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:17.633295 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:49:17.633215 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:17.733741 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:49:17.733715 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:21.590537 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:49:21.590498 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:49:24.245585 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:49:24.245549 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:28.825292 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:49:28.825259 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:30.829130 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:49:30.829054 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:31.312938 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:49:31.312904 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:31.394389 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:49:31.394355 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:49:51.591905 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:49:51.591866 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:50:03.475013 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:03.474964 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:03.576265 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:03.576228 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:18.909367 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:18.909335 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:19.747448 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:19.747412 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:20.542985 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:20.542951 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:21.590747 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:50:21.590711 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:50:22.869628 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:22.869590 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:23.717154 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:23.717120 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:24.123260 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:24.123212 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:24.436669 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:24.436592 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:26.768518 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:26.768478 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:28.072263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:28.072230 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-gvl5b"] Oct 30 00:50:28.075530 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:28.075510 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-gvl5b" Oct 30 00:50:28.105618 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:28.105587 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2rgw\" (UniqueName: \"kubernetes.io/projected/6bae3153-fe09-4948-9f0a-b6815c7f16e1-kube-api-access-r2rgw\") pod \"dns-operator-controller-manager-65bfd79598-gvl5b\" (UID: \"6bae3153-fe09-4948-9f0a-b6815c7f16e1\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-gvl5b" Oct 30 00:50:28.110702 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:28.110678 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 00:50:28.110702 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:28.110699 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 00:50:28.110702 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:28.110685 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 00:50:28.110985 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:28.110737 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-x8vbb\"" Oct 30 00:50:28.111457 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:28.111434 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-gvl5b"] Oct 30 00:50:28.206108 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:28.206074 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2rgw\" (UniqueName: \"kubernetes.io/projected/6bae3153-fe09-4948-9f0a-b6815c7f16e1-kube-api-access-r2rgw\") pod \"dns-operator-controller-manager-65bfd79598-gvl5b\" (UID: \"6bae3153-fe09-4948-9f0a-b6815c7f16e1\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-gvl5b" Oct 30 00:50:28.228209 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:28.228160 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2rgw\" (UniqueName: \"kubernetes.io/projected/6bae3153-fe09-4948-9f0a-b6815c7f16e1-kube-api-access-r2rgw\") pod \"dns-operator-controller-manager-65bfd79598-gvl5b\" (UID: \"6bae3153-fe09-4948-9f0a-b6815c7f16e1\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-gvl5b" Oct 30 00:50:28.387793 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:28.387666 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-gvl5b" Oct 30 00:50:28.562775 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:28.562730 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:28.684065 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:28.682587 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-gvl5b"] Oct 30 00:50:28.687871 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:50:28.687841 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bae3153_fe09_4948_9f0a_b6815c7f16e1.slice/crio-4e72fcb05ac157ca26b4b0671ac663ae8ba447b20b8cd7deddb41a79889d8944 WatchSource:0}: Error finding container 4e72fcb05ac157ca26b4b0671ac663ae8ba447b20b8cd7deddb41a79889d8944: Status 404 returned error can't find the container with id 4e72fcb05ac157ca26b4b0671ac663ae8ba447b20b8cd7deddb41a79889d8944 Oct 30 00:50:28.690711 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:28.690692 5300 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:50:28.734439 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:28.734393 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:28.961512 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:28.961405 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:29.330829 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:29.330793 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-gvl5b" event={"ID":"6bae3153-fe09-4948-9f0a-b6815c7f16e1","Type":"ContainerStarted","Data":"4e72fcb05ac157ca26b4b0671ac663ae8ba447b20b8cd7deddb41a79889d8944"} Oct 30 00:50:29.958816 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:29.958776 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:32.345423 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:32.345394 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-gvl5b" event={"ID":"6bae3153-fe09-4948-9f0a-b6815c7f16e1","Type":"ContainerStarted","Data":"bff0b08225a2af5f39d3871a362a1c120e775b500fe95f8ef2d0e0260fff7dd4"} Oct 30 00:50:32.345799 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:32.345496 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-gvl5b" Oct 30 00:50:32.384654 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:32.384609 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-gvl5b" podStartSLOduration=0.8574721 podStartE2EDuration="4.384594694s" totalImagesPullingTime="3.527122594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:32.383783486 +0000 UTC m=+432.699651805" watchObservedRunningTime="2026-10-30 00:50:32.384594694 +0000 UTC m=+432.700463015" Oct 30 00:50:33.910602 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:33.910565 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:34.015133 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:34.015098 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:34.309684 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:34.309652 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:34.409756 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:34.409727 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:34.715448 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:34.715379 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:34.800997 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:34.800966 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:42.726773 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:42.726736 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:42.802517 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:42.802485 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:43.351725 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:43.351696 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-gvl5b" Oct 30 00:50:45.165325 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:45.165297 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:45.266964 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:45.266933 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:49.932973 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:49.932943 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:50.033194 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:50.033164 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:51.590750 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:50:51.590713 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:50:51.916101 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:51.915985 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-rngkx"] Oct 30 00:50:51.918638 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:51.918607 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:51.919492 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:51.919473 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rngkx" Oct 30 00:50:51.934160 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:51.934137 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 00:50:51.934260 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:51.934163 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 00:50:51.934260 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:51.934218 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gs4jx\"" Oct 30 00:50:51.946846 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:51.946816 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-rngkx"] Oct 30 00:50:51.995602 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:51.995569 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84h4d\" (UniqueName: \"kubernetes.io/projected/17297963-0671-42e5-b81c-1dbf7e6ec6bd-kube-api-access-84h4d\") pod \"kuadrant-console-plugin-66c4fb575d-rngkx\" (UID: \"17297963-0671-42e5-b81c-1dbf7e6ec6bd\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rngkx" Oct 30 00:50:51.995746 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:51.995719 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/17297963-0671-42e5-b81c-1dbf7e6ec6bd-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-rngkx\" (UID: \"17297963-0671-42e5-b81c-1dbf7e6ec6bd\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rngkx" Oct 30 00:50:51.995827 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:51.995751 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/17297963-0671-42e5-b81c-1dbf7e6ec6bd-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-rngkx\" (UID: \"17297963-0671-42e5-b81c-1dbf7e6ec6bd\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rngkx" Oct 30 00:50:52.097041 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:52.097005 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/17297963-0671-42e5-b81c-1dbf7e6ec6bd-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-rngkx\" (UID: \"17297963-0671-42e5-b81c-1dbf7e6ec6bd\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rngkx" Oct 30 00:50:52.097041 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:52.097046 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/17297963-0671-42e5-b81c-1dbf7e6ec6bd-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-rngkx\" (UID: \"17297963-0671-42e5-b81c-1dbf7e6ec6bd\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rngkx" Oct 30 00:50:52.097278 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:52.097070 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84h4d\" (UniqueName: \"kubernetes.io/projected/17297963-0671-42e5-b81c-1dbf7e6ec6bd-kube-api-access-84h4d\") pod \"kuadrant-console-plugin-66c4fb575d-rngkx\" (UID: \"17297963-0671-42e5-b81c-1dbf7e6ec6bd\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rngkx" Oct 30 00:50:52.097653 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:52.097628 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/17297963-0671-42e5-b81c-1dbf7e6ec6bd-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-rngkx\" (UID: \"17297963-0671-42e5-b81c-1dbf7e6ec6bd\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rngkx" Oct 30 00:50:52.099520 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:52.099498 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/17297963-0671-42e5-b81c-1dbf7e6ec6bd-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-rngkx\" (UID: \"17297963-0671-42e5-b81c-1dbf7e6ec6bd\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rngkx" Oct 30 00:50:52.115806 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:52.115756 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84h4d\" (UniqueName: \"kubernetes.io/projected/17297963-0671-42e5-b81c-1dbf7e6ec6bd-kube-api-access-84h4d\") pod \"kuadrant-console-plugin-66c4fb575d-rngkx\" (UID: \"17297963-0671-42e5-b81c-1dbf7e6ec6bd\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rngkx" Oct 30 00:50:52.125968 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:52.125943 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:52.231555 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:52.231529 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rngkx" Oct 30 00:50:52.520637 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:52.520548 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-rngkx"] Oct 30 00:50:52.524715 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:50:52.524682 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17297963_0671_42e5_b81c_1dbf7e6ec6bd.slice/crio-6717bcc8e25e5092bd346dc925cac70a136152f949d180ddf4a687d94b7196e8 WatchSource:0}: Error finding container 6717bcc8e25e5092bd346dc925cac70a136152f949d180ddf4a687d94b7196e8: Status 404 returned error can't find the container with id 6717bcc8e25e5092bd346dc925cac70a136152f949d180ddf4a687d94b7196e8 Oct 30 00:50:53.425339 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:53.425310 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rngkx" event={"ID":"17297963-0671-42e5-b81c-1dbf7e6ec6bd","Type":"ContainerStarted","Data":"6717bcc8e25e5092bd346dc925cac70a136152f949d180ddf4a687d94b7196e8"} Oct 30 00:50:55.126636 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:55.126600 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:55.204723 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:55.204690 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:55.548733 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:55.548702 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:55.635718 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:55.635691 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:55.731940 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:55.731897 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:50:55.833060 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:50:55.832988 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:00.454522 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:00.454436 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rngkx" event={"ID":"17297963-0671-42e5-b81c-1dbf7e6ec6bd","Type":"ContainerStarted","Data":"9efb365df520d4a0193f6988e767bdf7ae42c28febb697ca77a66a2e00290d55"} Oct 30 00:51:00.503450 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:00.503401 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rngkx" podStartSLOduration=1.984963765 podStartE2EDuration="9.503388281s" totalImagesPullingTime="7.518424516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:00.502721812 +0000 UTC m=+460.818590136" watchObservedRunningTime="2026-10-30 00:51:00.503388281 +0000 UTC m=+460.819256597" Oct 30 00:51:01.147680 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:01.147601 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jqrdd"] Oct 30 00:51:01.151126 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:01.151105 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jqrdd" Oct 30 00:51:01.162419 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:01.162394 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jqrdd"] Oct 30 00:51:01.173938 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:01.173907 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-v6n6c\"" Oct 30 00:51:01.280385 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:01.280330 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rxz8\" (UniqueName: \"kubernetes.io/projected/842c6251-fea5-4613-913b-2a8387bf0888-kube-api-access-5rxz8\") pod \"kuadrant-operator-controller-manager-6b76784d48-jqrdd\" (UID: \"842c6251-fea5-4613-913b-2a8387bf0888\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jqrdd" Oct 30 00:51:01.280561 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:01.280436 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/842c6251-fea5-4613-913b-2a8387bf0888-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-jqrdd\" (UID: \"842c6251-fea5-4613-913b-2a8387bf0888\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jqrdd" Oct 30 00:51:01.381738 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:01.381700 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rxz8\" (UniqueName: \"kubernetes.io/projected/842c6251-fea5-4613-913b-2a8387bf0888-kube-api-access-5rxz8\") pod \"kuadrant-operator-controller-manager-6b76784d48-jqrdd\" (UID: \"842c6251-fea5-4613-913b-2a8387bf0888\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jqrdd" Oct 30 00:51:01.381922 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:01.381747 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/842c6251-fea5-4613-913b-2a8387bf0888-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-jqrdd\" (UID: \"842c6251-fea5-4613-913b-2a8387bf0888\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jqrdd" Oct 30 00:51:01.382126 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:01.382109 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/842c6251-fea5-4613-913b-2a8387bf0888-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-jqrdd\" (UID: \"842c6251-fea5-4613-913b-2a8387bf0888\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jqrdd" Oct 30 00:51:01.400725 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:01.400671 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rxz8\" (UniqueName: \"kubernetes.io/projected/842c6251-fea5-4613-913b-2a8387bf0888-kube-api-access-5rxz8\") pod \"kuadrant-operator-controller-manager-6b76784d48-jqrdd\" (UID: \"842c6251-fea5-4613-913b-2a8387bf0888\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jqrdd" Oct 30 00:51:01.463594 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:01.463567 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jqrdd" Oct 30 00:51:01.778852 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:01.778811 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jqrdd"] Oct 30 00:51:01.783043 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:51:01.783018 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842c6251_fea5_4613_913b_2a8387bf0888.slice/crio-ad4f6a53d5d6df8e2e1c9605d318ff854958e53b3c311d7f515b94ede371e119 WatchSource:0}: Error finding container ad4f6a53d5d6df8e2e1c9605d318ff854958e53b3c311d7f515b94ede371e119: Status 404 returned error can't find the container with id ad4f6a53d5d6df8e2e1c9605d318ff854958e53b3c311d7f515b94ede371e119 Oct 30 00:51:02.467156 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:02.467116 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jqrdd" event={"ID":"842c6251-fea5-4613-913b-2a8387bf0888","Type":"ContainerStarted","Data":"ad4f6a53d5d6df8e2e1c9605d318ff854958e53b3c311d7f515b94ede371e119"} Oct 30 00:51:03.345659 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:03.345619 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jqrdd"] Oct 30 00:51:03.346583 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:03.346521 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jqrdd"] Oct 30 00:51:03.383442 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:03.383362 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-7kpnp"] Oct 30 00:51:03.386601 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:03.386577 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-7kpnp" Oct 30 00:51:03.422056 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:03.421999 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-7kpnp"] Oct 30 00:51:03.499902 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:03.499871 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdhr8\" (UniqueName: \"kubernetes.io/projected/7ec82a6b-bed8-4f7f-a2ef-19819aef7c71-kube-api-access-rdhr8\") pod \"kuadrant-operator-controller-manager-6b76784d48-7kpnp\" (UID: \"7ec82a6b-bed8-4f7f-a2ef-19819aef7c71\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-7kpnp" Oct 30 00:51:03.500323 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:03.499932 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ec82a6b-bed8-4f7f-a2ef-19819aef7c71-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-7kpnp\" (UID: \"7ec82a6b-bed8-4f7f-a2ef-19819aef7c71\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-7kpnp" Oct 30 00:51:03.600949 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:03.600857 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdhr8\" (UniqueName: \"kubernetes.io/projected/7ec82a6b-bed8-4f7f-a2ef-19819aef7c71-kube-api-access-rdhr8\") pod \"kuadrant-operator-controller-manager-6b76784d48-7kpnp\" (UID: \"7ec82a6b-bed8-4f7f-a2ef-19819aef7c71\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-7kpnp" Oct 30 00:51:03.601125 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:03.600965 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ec82a6b-bed8-4f7f-a2ef-19819aef7c71-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-7kpnp\" (UID: \"7ec82a6b-bed8-4f7f-a2ef-19819aef7c71\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-7kpnp" Oct 30 00:51:03.601321 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:03.601305 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ec82a6b-bed8-4f7f-a2ef-19819aef7c71-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-7kpnp\" (UID: \"7ec82a6b-bed8-4f7f-a2ef-19819aef7c71\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-7kpnp" Oct 30 00:51:03.620408 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:03.620378 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdhr8\" (UniqueName: \"kubernetes.io/projected/7ec82a6b-bed8-4f7f-a2ef-19819aef7c71-kube-api-access-rdhr8\") pod \"kuadrant-operator-controller-manager-6b76784d48-7kpnp\" (UID: \"7ec82a6b-bed8-4f7f-a2ef-19819aef7c71\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-7kpnp" Oct 30 00:51:03.699348 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:03.699314 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-7kpnp" Oct 30 00:51:04.021747 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:04.021707 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-7kpnp"] Oct 30 00:51:04.038644 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:51:04.038611 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec82a6b_bed8_4f7f_a2ef_19819aef7c71.slice/crio-43c5c0bc7e3a661e493c01d266cbac501701c28552a938ac23c8a56462894c9c WatchSource:0}: Error finding container 43c5c0bc7e3a661e493c01d266cbac501701c28552a938ac23c8a56462894c9c: Status 404 returned error can't find the container with id 43c5c0bc7e3a661e493c01d266cbac501701c28552a938ac23c8a56462894c9c Oct 30 00:51:04.476444 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:04.476405 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-7kpnp" event={"ID":"7ec82a6b-bed8-4f7f-a2ef-19819aef7c71","Type":"ContainerStarted","Data":"43c5c0bc7e3a661e493c01d266cbac501701c28552a938ac23c8a56462894c9c"} Oct 30 00:51:06.332158 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:06.332118 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:06.411408 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:06.411374 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:06.927418 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:06.927353 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:07.013089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:07.013033 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:08.495695 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:08.495656 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jqrdd" podUID="842c6251-fea5-4613-913b-2a8387bf0888" containerName="manager" containerID="cri-o://14a7930e9b073bb69cd3aa844abc48f0bd6877f67eb5c1c79d6f7224695a7684" gracePeriod=2 Oct 30 00:51:08.497375 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:08.497345 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-7kpnp" event={"ID":"7ec82a6b-bed8-4f7f-a2ef-19819aef7c71","Type":"ContainerStarted","Data":"54a1eba985b99c9b9a6ae73c149fceb15fd67c9033e3cd2a158398e4347d10ca"} Oct 30 00:51:08.497498 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:08.497466 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-7kpnp" Oct 30 00:51:08.564408 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:08.564358 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-7kpnp" podStartSLOduration=1.92462744 podStartE2EDuration="5.56434396s" totalImagesPullingTime="3.63971652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:08.551929638 +0000 UTC m=+468.867797963" watchObservedRunningTime="2026-10-30 00:51:08.56434396 +0000 UTC m=+468.880212286" Oct 30 00:51:08.742978 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:08.742954 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jqrdd" Oct 30 00:51:08.757335 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:51:08.757270 5300 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-jqrdd\" is forbidden: User \"system:node:ip-10-0-140-206.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-206.ec2.internal' and this object" podUID="842c6251-fea5-4613-913b-2a8387bf0888" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jqrdd" Oct 30 00:51:08.845739 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:08.845702 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/842c6251-fea5-4613-913b-2a8387bf0888-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/842c6251-fea5-4613-913b-2a8387bf0888-extensions-socket-volume\") pod \"842c6251-fea5-4613-913b-2a8387bf0888\" (UID: \"842c6251-fea5-4613-913b-2a8387bf0888\") " Oct 30 00:51:08.845942 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:08.845797 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/842c6251-fea5-4613-913b-2a8387bf0888-kube-api-access-5rxz8\" (UniqueName: \"kubernetes.io/projected/842c6251-fea5-4613-913b-2a8387bf0888-kube-api-access-5rxz8\") pod \"842c6251-fea5-4613-913b-2a8387bf0888\" (UID: \"842c6251-fea5-4613-913b-2a8387bf0888\") " Oct 30 00:51:08.846029 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:08.846006 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/842c6251-fea5-4613-913b-2a8387bf0888-extensions-socket-volume" pod "842c6251-fea5-4613-913b-2a8387bf0888" (UID: "842c6251-fea5-4613-913b-2a8387bf0888"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:08.847969 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:08.847948 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842c6251-fea5-4613-913b-2a8387bf0888-kube-api-access-5rxz8" pod "842c6251-fea5-4613-913b-2a8387bf0888" (UID: "842c6251-fea5-4613-913b-2a8387bf0888"). InnerVolumeSpecName "kube-api-access-5rxz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:51:08.952397 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:08.947359 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rxz8\" (UniqueName: \"kubernetes.io/projected/842c6251-fea5-4613-913b-2a8387bf0888-kube-api-access-5rxz8\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:51:08.952397 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:08.947438 5300 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/842c6251-fea5-4613-913b-2a8387bf0888-extensions-socket-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:51:09.502165 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:09.502130 5300 generic.go:356] "Generic (PLEG): container finished" podID="842c6251-fea5-4613-913b-2a8387bf0888" containerID="14a7930e9b073bb69cd3aa844abc48f0bd6877f67eb5c1c79d6f7224695a7684" exitCode=2 Oct 30 00:51:09.502640 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:09.502175 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jqrdd" Oct 30 00:51:09.502640 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:09.502239 5300 scope.go:122] "RemoveContainer" containerID="14a7930e9b073bb69cd3aa844abc48f0bd6877f67eb5c1c79d6f7224695a7684" Oct 30 00:51:09.510295 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:09.510276 5300 scope.go:122] "RemoveContainer" containerID="14a7930e9b073bb69cd3aa844abc48f0bd6877f67eb5c1c79d6f7224695a7684" Oct 30 00:51:09.510592 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:51:09.510571 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a7930e9b073bb69cd3aa844abc48f0bd6877f67eb5c1c79d6f7224695a7684\": container with ID starting with 14a7930e9b073bb69cd3aa844abc48f0bd6877f67eb5c1c79d6f7224695a7684 not found: ID does not exist" containerID="14a7930e9b073bb69cd3aa844abc48f0bd6877f67eb5c1c79d6f7224695a7684" Oct 30 00:51:09.510677 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:09.510598 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a7930e9b073bb69cd3aa844abc48f0bd6877f67eb5c1c79d6f7224695a7684"} err="failed to get container status \"14a7930e9b073bb69cd3aa844abc48f0bd6877f67eb5c1c79d6f7224695a7684\": rpc error: code = NotFound desc = could not find container \"14a7930e9b073bb69cd3aa844abc48f0bd6877f67eb5c1c79d6f7224695a7684\": container with ID starting with 14a7930e9b073bb69cd3aa844abc48f0bd6877f67eb5c1c79d6f7224695a7684 not found: ID does not exist" Oct 30 00:51:09.515698 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:51:09.515676 5300 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-jqrdd\" is forbidden: User \"system:node:ip-10-0-140-206.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-206.ec2.internal' and this object" podUID="842c6251-fea5-4613-913b-2a8387bf0888" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jqrdd" Oct 30 00:51:09.529115 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:51:09.529090 5300 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-jqrdd\" is forbidden: User \"system:node:ip-10-0-140-206.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-206.ec2.internal' and this object" podUID="842c6251-fea5-4613-913b-2a8387bf0888" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jqrdd" Oct 30 00:51:09.818045 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:09.817973 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="842c6251-fea5-4613-913b-2a8387bf0888" path="/var/lib/kubelet/pods/842c6251-fea5-4613-913b-2a8387bf0888/volumes" Oct 30 00:51:09.827742 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:51:09.827712 5300 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-jqrdd\" is forbidden: User \"system:node:ip-10-0-140-206.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-206.ec2.internal' and this object" podUID="842c6251-fea5-4613-913b-2a8387bf0888" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-jqrdd" Oct 30 00:51:09.968896 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:09.968863 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:10.069318 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:10.069234 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:14.130964 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:14.130925 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:14.231095 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:14.231066 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:19.504833 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:19.504803 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-7kpnp" Oct 30 00:51:21.592116 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:51:21.592080 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:51:29.057638 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:29.057584 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:32.078355 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:32.078317 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-7kpnp"] Oct 30 00:51:32.078819 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:32.078541 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-7kpnp" podUID="7ec82a6b-bed8-4f7f-a2ef-19819aef7c71" containerName="manager" containerID="cri-o://54a1eba985b99c9b9a6ae73c149fceb15fd67c9033e3cd2a158398e4347d10ca" gracePeriod=10 Oct 30 00:51:32.595339 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:32.595307 5300 generic.go:356] "Generic (PLEG): container finished" podID="7ec82a6b-bed8-4f7f-a2ef-19819aef7c71" containerID="54a1eba985b99c9b9a6ae73c149fceb15fd67c9033e3cd2a158398e4347d10ca" exitCode=0 Oct 30 00:51:32.595503 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:32.595349 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-7kpnp" event={"ID":"7ec82a6b-bed8-4f7f-a2ef-19819aef7c71","Type":"ContainerDied","Data":"54a1eba985b99c9b9a6ae73c149fceb15fd67c9033e3cd2a158398e4347d10ca"} Oct 30 00:51:32.814986 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:32.814958 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-7kpnp" Oct 30 00:51:32.935251 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:32.935175 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ec82a6b-bed8-4f7f-a2ef-19819aef7c71-kube-api-access-rdhr8\" (UniqueName: \"kubernetes.io/projected/7ec82a6b-bed8-4f7f-a2ef-19819aef7c71-kube-api-access-rdhr8\") pod \"7ec82a6b-bed8-4f7f-a2ef-19819aef7c71\" (UID: \"7ec82a6b-bed8-4f7f-a2ef-19819aef7c71\") " Oct 30 00:51:32.935251 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:32.935236 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec82a6b-bed8-4f7f-a2ef-19819aef7c71-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ec82a6b-bed8-4f7f-a2ef-19819aef7c71-extensions-socket-volume\") pod \"7ec82a6b-bed8-4f7f-a2ef-19819aef7c71\" (UID: \"7ec82a6b-bed8-4f7f-a2ef-19819aef7c71\") " Oct 30 00:51:32.935634 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:32.935608 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec82a6b-bed8-4f7f-a2ef-19819aef7c71-extensions-socket-volume" pod "7ec82a6b-bed8-4f7f-a2ef-19819aef7c71" (UID: "7ec82a6b-bed8-4f7f-a2ef-19819aef7c71"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 00:51:32.937268 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:32.937249 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec82a6b-bed8-4f7f-a2ef-19819aef7c71-kube-api-access-rdhr8" pod "7ec82a6b-bed8-4f7f-a2ef-19819aef7c71" (UID: "7ec82a6b-bed8-4f7f-a2ef-19819aef7c71"). InnerVolumeSpecName "kube-api-access-rdhr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:51:33.035821 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:33.035787 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdhr8\" (UniqueName: \"kubernetes.io/projected/7ec82a6b-bed8-4f7f-a2ef-19819aef7c71-kube-api-access-rdhr8\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:51:33.035821 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:33.035814 5300 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ec82a6b-bed8-4f7f-a2ef-19819aef7c71-extensions-socket-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:51:33.600082 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:33.600048 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-7kpnp" event={"ID":"7ec82a6b-bed8-4f7f-a2ef-19819aef7c71","Type":"ContainerDied","Data":"43c5c0bc7e3a661e493c01d266cbac501701c28552a938ac23c8a56462894c9c"} Oct 30 00:51:33.600082 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:33.600067 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-7kpnp" Oct 30 00:51:33.600527 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:33.600111 5300 scope.go:122] "RemoveContainer" containerID="54a1eba985b99c9b9a6ae73c149fceb15fd67c9033e3cd2a158398e4347d10ca" Oct 30 00:51:33.661019 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:33.660974 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-7kpnp"] Oct 30 00:51:33.673212 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:33.673180 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-7kpnp"] Oct 30 00:51:33.817608 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:33.817572 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ec82a6b-bed8-4f7f-a2ef-19819aef7c71" path="/var/lib/kubelet/pods/7ec82a6b-bed8-4f7f-a2ef-19819aef7c71/volumes" Oct 30 00:51:36.700818 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:36.700785 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:36.892167 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:36.892136 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:50.980338 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:50.980300 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:50.993592 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:50.993570 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7xff8"] Oct 30 00:51:50.993938 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:50.993907 5300 cpu_manager.go:414] "containerMap: removing container" podUID="7ec82a6b-bed8-4f7f-a2ef-19819aef7c71" containerName="manager" Oct 30 00:51:50.993938 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:50.993930 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ec82a6b-bed8-4f7f-a2ef-19819aef7c71" containerName="manager" Oct 30 00:51:50.994069 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:50.993943 5300 cpu_manager.go:414] "containerMap: removing container" podUID="842c6251-fea5-4613-913b-2a8387bf0888" containerName="manager" Oct 30 00:51:50.994069 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:50.993948 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="842c6251-fea5-4613-913b-2a8387bf0888" containerName="manager" Oct 30 00:51:50.994069 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:50.994019 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="842c6251-fea5-4613-913b-2a8387bf0888" containerName="manager" Oct 30 00:51:50.994069 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:50.994037 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ec82a6b-bed8-4f7f-a2ef-19819aef7c71" containerName="manager" Oct 30 00:51:50.998422 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:50.998405 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-7xff8" Oct 30 00:51:51.014506 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:51.014478 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 00:51:51.015366 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:51.015346 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7xff8"] Oct 30 00:51:51.086381 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:51.086348 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j69ht\" (UniqueName: \"kubernetes.io/projected/e086d307-7957-42bf-aa8e-a4df36153837-kube-api-access-j69ht\") pod \"limitador-limitador-74dfcfb894-7xff8\" (UID: \"e086d307-7957-42bf-aa8e-a4df36153837\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-7xff8" Oct 30 00:51:51.086538 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:51.086402 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e086d307-7957-42bf-aa8e-a4df36153837-config-file\") pod \"limitador-limitador-74dfcfb894-7xff8\" (UID: \"e086d307-7957-42bf-aa8e-a4df36153837\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-7xff8" Oct 30 00:51:51.102500 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:51.102473 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:51.123583 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:51.123550 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7xff8"] Oct 30 00:51:51.187216 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:51.187172 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e086d307-7957-42bf-aa8e-a4df36153837-config-file\") pod \"limitador-limitador-74dfcfb894-7xff8\" (UID: \"e086d307-7957-42bf-aa8e-a4df36153837\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-7xff8" Oct 30 00:51:51.187379 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:51.187246 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j69ht\" (UniqueName: \"kubernetes.io/projected/e086d307-7957-42bf-aa8e-a4df36153837-kube-api-access-j69ht\") pod \"limitador-limitador-74dfcfb894-7xff8\" (UID: \"e086d307-7957-42bf-aa8e-a4df36153837\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-7xff8" Oct 30 00:51:51.187828 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:51.187806 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e086d307-7957-42bf-aa8e-a4df36153837-config-file\") pod \"limitador-limitador-74dfcfb894-7xff8\" (UID: \"e086d307-7957-42bf-aa8e-a4df36153837\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-7xff8" Oct 30 00:51:51.200399 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:51.200373 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:51.205567 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:51.205540 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j69ht\" (UniqueName: \"kubernetes.io/projected/e086d307-7957-42bf-aa8e-a4df36153837-kube-api-access-j69ht\") pod \"limitador-limitador-74dfcfb894-7xff8\" (UID: \"e086d307-7957-42bf-aa8e-a4df36153837\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-7xff8" Oct 30 00:51:51.310862 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:51.310753 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-7xff8" Oct 30 00:51:51.591517 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:51:51.591430 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:51:51.597408 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:51.597385 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7xff8"] Oct 30 00:51:51.602228 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:51:51.602205 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode086d307_7957_42bf_aa8e_a4df36153837.slice/crio-2fdccb159e18caa92caf313123ee31cccb1996165ba2e1788d0a5ae50b91856a WatchSource:0}: Error finding container 2fdccb159e18caa92caf313123ee31cccb1996165ba2e1788d0a5ae50b91856a: Status 404 returned error can't find the container with id 2fdccb159e18caa92caf313123ee31cccb1996165ba2e1788d0a5ae50b91856a Oct 30 00:51:51.668558 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:51.668520 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-7xff8" event={"ID":"e086d307-7957-42bf-aa8e-a4df36153837","Type":"ContainerStarted","Data":"2fdccb159e18caa92caf313123ee31cccb1996165ba2e1788d0a5ae50b91856a"} Oct 30 00:51:51.848815 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:51.848716 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5976w"] Oct 30 00:51:51.853127 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:51.853105 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-5976w" Oct 30 00:51:51.867116 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:51.867097 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5976w"] Oct 30 00:51:51.878537 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:51.878518 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5vnz2\"" Oct 30 00:51:51.914564 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:51.914534 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:51.994562 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:51.994527 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hm62\" (UniqueName: \"kubernetes.io/projected/626295e6-8f20-4ca3-8e20-ab229ff9fc47-kube-api-access-9hm62\") pod \"authorino-57f7f8659f-5976w\" (UID: \"626295e6-8f20-4ca3-8e20-ab229ff9fc47\") " pod="kuadrant-system/authorino-57f7f8659f-5976w" Oct 30 00:51:52.016029 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:52.015999 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:51:52.095005 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:52.094965 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hm62\" (UniqueName: \"kubernetes.io/projected/626295e6-8f20-4ca3-8e20-ab229ff9fc47-kube-api-access-9hm62\") pod \"authorino-57f7f8659f-5976w\" (UID: \"626295e6-8f20-4ca3-8e20-ab229ff9fc47\") " pod="kuadrant-system/authorino-57f7f8659f-5976w" Oct 30 00:51:52.118160 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:52.118101 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hm62\" (UniqueName: \"kubernetes.io/projected/626295e6-8f20-4ca3-8e20-ab229ff9fc47-kube-api-access-9hm62\") pod \"authorino-57f7f8659f-5976w\" (UID: \"626295e6-8f20-4ca3-8e20-ab229ff9fc47\") " pod="kuadrant-system/authorino-57f7f8659f-5976w" Oct 30 00:51:52.163996 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:52.163971 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-5976w" Oct 30 00:51:52.429797 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:52.429704 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-pjfjd"] Oct 30 00:51:52.433140 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:52.433118 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-pjfjd" Oct 30 00:51:52.452472 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:52.452449 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-pjfjd"] Oct 30 00:51:52.458680 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:52.458635 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5976w"] Oct 30 00:51:52.463193 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:51:52.463167 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod626295e6_8f20_4ca3_8e20_ab229ff9fc47.slice/crio-501e261fb8dc3404da2fd53c7dcc3a5ddbd08fe19739582a2a379a268d2fc2e7 WatchSource:0}: Error finding container 501e261fb8dc3404da2fd53c7dcc3a5ddbd08fe19739582a2a379a268d2fc2e7: Status 404 returned error can't find the container with id 501e261fb8dc3404da2fd53c7dcc3a5ddbd08fe19739582a2a379a268d2fc2e7 Oct 30 00:51:52.498467 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:52.498436 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22jjg\" (UniqueName: \"kubernetes.io/projected/55d6773b-3df4-4a4a-bb3e-d895fdf8397c-kube-api-access-22jjg\") pod \"authorino-56c6c9ddd5-pjfjd\" (UID: \"55d6773b-3df4-4a4a-bb3e-d895fdf8397c\") " pod="kuadrant-system/authorino-56c6c9ddd5-pjfjd" Oct 30 00:51:52.599838 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:52.599795 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22jjg\" (UniqueName: \"kubernetes.io/projected/55d6773b-3df4-4a4a-bb3e-d895fdf8397c-kube-api-access-22jjg\") pod \"authorino-56c6c9ddd5-pjfjd\" (UID: \"55d6773b-3df4-4a4a-bb3e-d895fdf8397c\") " pod="kuadrant-system/authorino-56c6c9ddd5-pjfjd" Oct 30 00:51:52.618565 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:52.618538 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22jjg\" (UniqueName: \"kubernetes.io/projected/55d6773b-3df4-4a4a-bb3e-d895fdf8397c-kube-api-access-22jjg\") pod \"authorino-56c6c9ddd5-pjfjd\" (UID: \"55d6773b-3df4-4a4a-bb3e-d895fdf8397c\") " pod="kuadrant-system/authorino-56c6c9ddd5-pjfjd" Oct 30 00:51:52.673462 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:52.673432 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-5976w" event={"ID":"626295e6-8f20-4ca3-8e20-ab229ff9fc47","Type":"ContainerStarted","Data":"501e261fb8dc3404da2fd53c7dcc3a5ddbd08fe19739582a2a379a268d2fc2e7"} Oct 30 00:51:52.746642 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:52.746611 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-pjfjd" Oct 30 00:51:53.036970 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:53.036887 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-pjfjd"] Oct 30 00:51:53.041110 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:51:53.041080 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55d6773b_3df4_4a4a_bb3e_d895fdf8397c.slice/crio-ceac2ee00d69edacefff733d268f127d0673280e9ce139ea838a16f629ceca12 WatchSource:0}: Error finding container ceac2ee00d69edacefff733d268f127d0673280e9ce139ea838a16f629ceca12: Status 404 returned error can't find the container with id ceac2ee00d69edacefff733d268f127d0673280e9ce139ea838a16f629ceca12 Oct 30 00:51:53.680269 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:53.680233 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-7xff8" event={"ID":"e086d307-7957-42bf-aa8e-a4df36153837","Type":"ContainerStarted","Data":"96e3218a129d6a638d649d67e6b45aa84301525b2e3587c25845141563dbd47f"} Oct 30 00:51:53.680475 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:53.680428 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-7xff8" Oct 30 00:51:53.681797 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:53.681747 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-pjfjd" event={"ID":"55d6773b-3df4-4a4a-bb3e-d895fdf8397c","Type":"ContainerStarted","Data":"ceac2ee00d69edacefff733d268f127d0673280e9ce139ea838a16f629ceca12"} Oct 30 00:51:53.719384 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:53.717736 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-7xff8" podStartSLOduration=2.589627222 podStartE2EDuration="3.717717715s" totalImagesPullingTime="1.128090493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:53.71684545 +0000 UTC m=+514.032713776" watchObservedRunningTime="2026-10-30 00:51:53.717717715 +0000 UTC m=+514.033586041" Oct 30 00:51:54.688162 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:54.688125 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-5976w" event={"ID":"626295e6-8f20-4ca3-8e20-ab229ff9fc47","Type":"ContainerStarted","Data":"9cb8a889a52f766c859e98d376782ad5f6940747ff035b654a732f30a7b425b2"} Oct 30 00:51:54.689905 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:54.689879 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-pjfjd" event={"ID":"55d6773b-3df4-4a4a-bb3e-d895fdf8397c","Type":"ContainerStarted","Data":"c4f757de61016fed68e3f2a38bfaa00d57cc93e7eb479e7872dadab7c8f81813"} Oct 30 00:51:54.724474 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:54.724402 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-5976w" podStartSLOduration=1.730881634 podStartE2EDuration="3.724387514s" totalImagesPullingTime="1.99350588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:54.723416268 +0000 UTC m=+515.039284592" watchObservedRunningTime="2026-10-30 00:51:54.724387514 +0000 UTC m=+515.040255839" Oct 30 00:51:54.761002 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:54.760889 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-pjfjd" podStartSLOduration=1.273932448 podStartE2EDuration="2.760870146s" totalImagesPullingTime="1.486937698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:54.75983957 +0000 UTC m=+515.075707895" watchObservedRunningTime="2026-10-30 00:51:54.760870146 +0000 UTC m=+515.076738474" Oct 30 00:51:54.780575 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:54.780543 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5976w"] Oct 30 00:51:56.697404 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:56.697356 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-5976w" podUID="626295e6-8f20-4ca3-8e20-ab229ff9fc47" containerName="authorino" containerID="cri-o://9cb8a889a52f766c859e98d376782ad5f6940747ff035b654a732f30a7b425b2" gracePeriod=30 Oct 30 00:51:56.930340 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:56.930308 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-5976w" Oct 30 00:51:57.038616 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:57.038579 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/626295e6-8f20-4ca3-8e20-ab229ff9fc47-kube-api-access-9hm62\" (UniqueName: \"kubernetes.io/projected/626295e6-8f20-4ca3-8e20-ab229ff9fc47-kube-api-access-9hm62\") pod \"626295e6-8f20-4ca3-8e20-ab229ff9fc47\" (UID: \"626295e6-8f20-4ca3-8e20-ab229ff9fc47\") " Oct 30 00:51:57.040722 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:57.040687 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/626295e6-8f20-4ca3-8e20-ab229ff9fc47-kube-api-access-9hm62" pod "626295e6-8f20-4ca3-8e20-ab229ff9fc47" (UID: "626295e6-8f20-4ca3-8e20-ab229ff9fc47"). InnerVolumeSpecName "kube-api-access-9hm62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:51:57.139432 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:57.139375 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hm62\" (UniqueName: \"kubernetes.io/projected/626295e6-8f20-4ca3-8e20-ab229ff9fc47-kube-api-access-9hm62\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:51:57.702617 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:57.702582 5300 generic.go:356] "Generic (PLEG): container finished" podID="626295e6-8f20-4ca3-8e20-ab229ff9fc47" containerID="9cb8a889a52f766c859e98d376782ad5f6940747ff035b654a732f30a7b425b2" exitCode=0 Oct 30 00:51:57.703096 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:57.702624 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-5976w" Oct 30 00:51:57.703096 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:57.702658 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-5976w" event={"ID":"626295e6-8f20-4ca3-8e20-ab229ff9fc47","Type":"ContainerDied","Data":"9cb8a889a52f766c859e98d376782ad5f6940747ff035b654a732f30a7b425b2"} Oct 30 00:51:57.703096 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:57.702696 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-5976w" event={"ID":"626295e6-8f20-4ca3-8e20-ab229ff9fc47","Type":"ContainerDied","Data":"501e261fb8dc3404da2fd53c7dcc3a5ddbd08fe19739582a2a379a268d2fc2e7"} Oct 30 00:51:57.703096 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:57.702719 5300 scope.go:122] "RemoveContainer" containerID="9cb8a889a52f766c859e98d376782ad5f6940747ff035b654a732f30a7b425b2" Oct 30 00:51:57.710707 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:57.710687 5300 scope.go:122] "RemoveContainer" containerID="9cb8a889a52f766c859e98d376782ad5f6940747ff035b654a732f30a7b425b2" Oct 30 00:51:57.710975 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:51:57.710958 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cb8a889a52f766c859e98d376782ad5f6940747ff035b654a732f30a7b425b2\": container with ID starting with 9cb8a889a52f766c859e98d376782ad5f6940747ff035b654a732f30a7b425b2 not found: ID does not exist" containerID="9cb8a889a52f766c859e98d376782ad5f6940747ff035b654a732f30a7b425b2" Oct 30 00:51:57.711045 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:57.710986 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cb8a889a52f766c859e98d376782ad5f6940747ff035b654a732f30a7b425b2"} err="failed to get container status \"9cb8a889a52f766c859e98d376782ad5f6940747ff035b654a732f30a7b425b2\": rpc error: code = NotFound desc = could not find container \"9cb8a889a52f766c859e98d376782ad5f6940747ff035b654a732f30a7b425b2\": container with ID starting with 9cb8a889a52f766c859e98d376782ad5f6940747ff035b654a732f30a7b425b2 not found: ID does not exist" Oct 30 00:51:57.749930 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:57.749864 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5976w"] Oct 30 00:51:57.761850 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:57.761820 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5976w"] Oct 30 00:51:57.818452 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:51:57.818419 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="626295e6-8f20-4ca3-8e20-ab229ff9fc47" path="/var/lib/kubelet/pods/626295e6-8f20-4ca3-8e20-ab229ff9fc47/volumes" Oct 30 00:52:04.691631 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:04.691600 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-7xff8" Oct 30 00:52:05.188426 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:05.188338 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7xff8"] Oct 30 00:52:05.188630 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:05.188607 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-7xff8" podUID="e086d307-7957-42bf-aa8e-a4df36153837" containerName="limitador" containerID="cri-o://96e3218a129d6a638d649d67e6b45aa84301525b2e3587c25845141563dbd47f" gracePeriod=30 Oct 30 00:52:05.742383 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:05.742353 5300 generic.go:356] "Generic (PLEG): container finished" podID="e086d307-7957-42bf-aa8e-a4df36153837" containerID="96e3218a129d6a638d649d67e6b45aa84301525b2e3587c25845141563dbd47f" exitCode=0 Oct 30 00:52:05.742772 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:05.742406 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-7xff8" event={"ID":"e086d307-7957-42bf-aa8e-a4df36153837","Type":"ContainerDied","Data":"96e3218a129d6a638d649d67e6b45aa84301525b2e3587c25845141563dbd47f"} Oct 30 00:52:06.119826 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:06.119804 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-7xff8" Oct 30 00:52:06.215332 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:06.215304 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e086d307-7957-42bf-aa8e-a4df36153837-config-file\" (UniqueName: \"kubernetes.io/configmap/e086d307-7957-42bf-aa8e-a4df36153837-config-file\") pod \"e086d307-7957-42bf-aa8e-a4df36153837\" (UID: \"e086d307-7957-42bf-aa8e-a4df36153837\") " Oct 30 00:52:06.215498 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:06.215381 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e086d307-7957-42bf-aa8e-a4df36153837-kube-api-access-j69ht\" (UniqueName: \"kubernetes.io/projected/e086d307-7957-42bf-aa8e-a4df36153837-kube-api-access-j69ht\") pod \"e086d307-7957-42bf-aa8e-a4df36153837\" (UID: \"e086d307-7957-42bf-aa8e-a4df36153837\") " Oct 30 00:52:06.215658 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:06.215636 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e086d307-7957-42bf-aa8e-a4df36153837-config-file" pod "e086d307-7957-42bf-aa8e-a4df36153837" (UID: "e086d307-7957-42bf-aa8e-a4df36153837"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 00:52:06.217506 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:06.217488 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e086d307-7957-42bf-aa8e-a4df36153837-kube-api-access-j69ht" pod "e086d307-7957-42bf-aa8e-a4df36153837" (UID: "e086d307-7957-42bf-aa8e-a4df36153837"). InnerVolumeSpecName "kube-api-access-j69ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:52:06.316986 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:06.316904 5300 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e086d307-7957-42bf-aa8e-a4df36153837-config-file\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:52:06.316986 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:06.316933 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j69ht\" (UniqueName: \"kubernetes.io/projected/e086d307-7957-42bf-aa8e-a4df36153837-kube-api-access-j69ht\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:52:06.747537 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:06.747503 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-7xff8" event={"ID":"e086d307-7957-42bf-aa8e-a4df36153837","Type":"ContainerDied","Data":"2fdccb159e18caa92caf313123ee31cccb1996165ba2e1788d0a5ae50b91856a"} Oct 30 00:52:06.747537 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:06.747528 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-7xff8" Oct 30 00:52:06.748063 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:06.747560 5300 scope.go:122] "RemoveContainer" containerID="96e3218a129d6a638d649d67e6b45aa84301525b2e3587c25845141563dbd47f" Oct 30 00:52:06.796050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:06.796016 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7xff8"] Oct 30 00:52:06.808110 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:06.808078 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-7xff8"] Oct 30 00:52:07.818092 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:07.818059 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e086d307-7957-42bf-aa8e-a4df36153837" path="/var/lib/kubelet/pods/e086d307-7957-42bf-aa8e-a4df36153837/volumes" Oct 30 00:52:11.729018 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:11.728983 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:12.413669 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:12.413631 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-5nb97"] Oct 30 00:52:12.414062 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:12.414034 5300 cpu_manager.go:414] "containerMap: removing container" podUID="e086d307-7957-42bf-aa8e-a4df36153837" containerName="limitador" Oct 30 00:52:12.414062 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:12.414059 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e086d307-7957-42bf-aa8e-a4df36153837" containerName="limitador" Oct 30 00:52:12.414216 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:12.414075 5300 cpu_manager.go:414] "containerMap: removing container" podUID="626295e6-8f20-4ca3-8e20-ab229ff9fc47" containerName="authorino" Oct 30 00:52:12.414216 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:12.414082 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="626295e6-8f20-4ca3-8e20-ab229ff9fc47" containerName="authorino" Oct 30 00:52:12.414216 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:12.414177 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="626295e6-8f20-4ca3-8e20-ab229ff9fc47" containerName="authorino" Oct 30 00:52:12.414216 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:12.414191 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="e086d307-7957-42bf-aa8e-a4df36153837" containerName="limitador" Oct 30 00:52:12.417044 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:12.417027 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5nb97" Oct 30 00:52:12.434639 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:12.434615 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-5nb97"] Oct 30 00:52:12.445988 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:12.445960 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 00:52:12.446141 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:12.446029 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 00:52:12.446141 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:12.446050 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 00:52:12.446141 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:12.446084 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-r9rll\"" Oct 30 00:52:12.566091 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:12.566056 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4907aff2-e13d-41bc-92c9-6f9a522a81c4-data\") pod \"postgres-7cdbf6dc44-5nb97\" (UID: \"4907aff2-e13d-41bc-92c9-6f9a522a81c4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5nb97" Oct 30 00:52:12.566263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:12.566192 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpfgz\" (UniqueName: \"kubernetes.io/projected/4907aff2-e13d-41bc-92c9-6f9a522a81c4-kube-api-access-mpfgz\") pod \"postgres-7cdbf6dc44-5nb97\" (UID: \"4907aff2-e13d-41bc-92c9-6f9a522a81c4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5nb97" Oct 30 00:52:12.667414 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:12.667330 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpfgz\" (UniqueName: \"kubernetes.io/projected/4907aff2-e13d-41bc-92c9-6f9a522a81c4-kube-api-access-mpfgz\") pod \"postgres-7cdbf6dc44-5nb97\" (UID: \"4907aff2-e13d-41bc-92c9-6f9a522a81c4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5nb97" Oct 30 00:52:12.667414 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:12.667371 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4907aff2-e13d-41bc-92c9-6f9a522a81c4-data\") pod \"postgres-7cdbf6dc44-5nb97\" (UID: \"4907aff2-e13d-41bc-92c9-6f9a522a81c4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5nb97" Oct 30 00:52:12.667720 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:12.667701 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4907aff2-e13d-41bc-92c9-6f9a522a81c4-data\") pod \"postgres-7cdbf6dc44-5nb97\" (UID: \"4907aff2-e13d-41bc-92c9-6f9a522a81c4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5nb97" Oct 30 00:52:12.685812 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:12.685786 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpfgz\" (UniqueName: \"kubernetes.io/projected/4907aff2-e13d-41bc-92c9-6f9a522a81c4-kube-api-access-mpfgz\") pod \"postgres-7cdbf6dc44-5nb97\" (UID: \"4907aff2-e13d-41bc-92c9-6f9a522a81c4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5nb97" Oct 30 00:52:12.728667 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:12.728633 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5nb97" Oct 30 00:52:13.019325 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:13.019290 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-5nb97"] Oct 30 00:52:13.023327 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:52:13.023298 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4907aff2_e13d_41bc_92c9_6f9a522a81c4.slice/crio-8a9b0fef74156096dd45bcfe288bd61238f3d270aa4edb7f6f0ec32429f6d49f WatchSource:0}: Error finding container 8a9b0fef74156096dd45bcfe288bd61238f3d270aa4edb7f6f0ec32429f6d49f: Status 404 returned error can't find the container with id 8a9b0fef74156096dd45bcfe288bd61238f3d270aa4edb7f6f0ec32429f6d49f Oct 30 00:52:13.564221 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:13.564190 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:13.778069 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:13.778034 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5nb97" event={"ID":"4907aff2-e13d-41bc-92c9-6f9a522a81c4","Type":"ContainerStarted","Data":"8a9b0fef74156096dd45bcfe288bd61238f3d270aa4edb7f6f0ec32429f6d49f"} Oct 30 00:52:18.801363 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:18.801322 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5nb97" event={"ID":"4907aff2-e13d-41bc-92c9-6f9a522a81c4","Type":"ContainerStarted","Data":"05d6bd139a87ca2928673a9afea79654941438d430c5ee56d1cd13e0ca7d16b9"} Oct 30 00:52:18.801745 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:18.801432 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5nb97" Oct 30 00:52:18.838563 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:18.838512 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5nb97" podStartSLOduration=1.799423668 podStartE2EDuration="6.838499128s" totalImagesPullingTime="5.03907546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:18.837900072 +0000 UTC m=+539.153768398" watchObservedRunningTime="2026-10-30 00:52:18.838499128 +0000 UTC m=+539.154367453" Oct 30 00:52:21.591381 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:52:21.591337 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:52:24.834229 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:24.834202 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5nb97" Oct 30 00:52:25.259251 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:25.259216 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:25.485410 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:25.485382 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:25.572945 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:25.572876 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-brzhj"] Oct 30 00:52:25.582439 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:25.582413 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-brzhj" Oct 30 00:52:25.593045 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:25.593020 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-brzhj"] Oct 30 00:52:25.660905 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:25.660877 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbdvg\" (UniqueName: \"kubernetes.io/projected/46878bd8-5b05-4220-9f14-dea40c76cc8e-kube-api-access-fbdvg\") pod \"authorino-6f6944bcdd-brzhj\" (UID: \"46878bd8-5b05-4220-9f14-dea40c76cc8e\") " pod="kuadrant-system/authorino-6f6944bcdd-brzhj" Oct 30 00:52:25.761513 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:25.761486 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbdvg\" (UniqueName: \"kubernetes.io/projected/46878bd8-5b05-4220-9f14-dea40c76cc8e-kube-api-access-fbdvg\") pod \"authorino-6f6944bcdd-brzhj\" (UID: \"46878bd8-5b05-4220-9f14-dea40c76cc8e\") " pod="kuadrant-system/authorino-6f6944bcdd-brzhj" Oct 30 00:52:25.779909 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:25.779883 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbdvg\" (UniqueName: \"kubernetes.io/projected/46878bd8-5b05-4220-9f14-dea40c76cc8e-kube-api-access-fbdvg\") pod \"authorino-6f6944bcdd-brzhj\" (UID: \"46878bd8-5b05-4220-9f14-dea40c76cc8e\") " pod="kuadrant-system/authorino-6f6944bcdd-brzhj" Oct 30 00:52:25.862371 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:25.862299 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-brzhj"] Oct 30 00:52:25.862857 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:25.862495 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-brzhj" Oct 30 00:52:25.887218 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:25.887194 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-t7l57"] Oct 30 00:52:25.897307 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:25.897281 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-t7l57" Oct 30 00:52:25.908260 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:25.907732 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-t7l57"] Oct 30 00:52:25.916166 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:25.916145 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 00:52:25.929095 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:25.929074 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-t7l57"] Oct 30 00:52:25.958852 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:25.956889 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dfd97fb6f-4rk72"] Oct 30 00:52:25.964899 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:25.963452 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7dcae54-a627-4094-a940-3c5fb56d4bec-tls-cert\") pod \"authorino-6889cbd8c4-t7l57\" (UID: \"c7dcae54-a627-4094-a940-3c5fb56d4bec\") " pod="kuadrant-system/authorino-6889cbd8c4-t7l57" Oct 30 00:52:25.964899 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:25.963513 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmhbv\" (UniqueName: \"kubernetes.io/projected/c7dcae54-a627-4094-a940-3c5fb56d4bec-kube-api-access-tmhbv\") pod \"authorino-6889cbd8c4-t7l57\" (UID: \"c7dcae54-a627-4094-a940-3c5fb56d4bec\") " pod="kuadrant-system/authorino-6889cbd8c4-t7l57" Oct 30 00:52:25.975672 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:52:25.975630 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-tmhbv tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-t7l57" podUID="c7dcae54-a627-4094-a940-3c5fb56d4bec" Oct 30 00:52:25.993073 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:25.993048 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dfd97fb6f-4rk72"] Oct 30 00:52:25.993184 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:25.993149 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dfd97fb6f-4rk72" Oct 30 00:52:26.064586 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:26.064553 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7dcae54-a627-4094-a940-3c5fb56d4bec-tls-cert\") pod \"authorino-6889cbd8c4-t7l57\" (UID: \"c7dcae54-a627-4094-a940-3c5fb56d4bec\") " pod="kuadrant-system/authorino-6889cbd8c4-t7l57" Oct 30 00:52:26.064729 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:26.064592 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/839c0725-66a2-4100-bd25-da52516a001c-tls-cert\") pod \"authorino-5dfd97fb6f-4rk72\" (UID: \"839c0725-66a2-4100-bd25-da52516a001c\") " pod="kuadrant-system/authorino-5dfd97fb6f-4rk72" Oct 30 00:52:26.064729 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:26.064617 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmhbv\" (UniqueName: \"kubernetes.io/projected/c7dcae54-a627-4094-a940-3c5fb56d4bec-kube-api-access-tmhbv\") pod \"authorino-6889cbd8c4-t7l57\" (UID: \"c7dcae54-a627-4094-a940-3c5fb56d4bec\") " pod="kuadrant-system/authorino-6889cbd8c4-t7l57" Oct 30 00:52:26.064729 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:26.064636 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzp8f\" (UniqueName: \"kubernetes.io/projected/839c0725-66a2-4100-bd25-da52516a001c-kube-api-access-wzp8f\") pod \"authorino-5dfd97fb6f-4rk72\" (UID: \"839c0725-66a2-4100-bd25-da52516a001c\") " pod="kuadrant-system/authorino-5dfd97fb6f-4rk72" Oct 30 00:52:26.066960 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:26.066938 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7dcae54-a627-4094-a940-3c5fb56d4bec-tls-cert\") pod \"authorino-6889cbd8c4-t7l57\" (UID: \"c7dcae54-a627-4094-a940-3c5fb56d4bec\") " pod="kuadrant-system/authorino-6889cbd8c4-t7l57" Oct 30 00:52:26.083356 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:26.083331 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmhbv\" (UniqueName: \"kubernetes.io/projected/c7dcae54-a627-4094-a940-3c5fb56d4bec-kube-api-access-tmhbv\") pod \"authorino-6889cbd8c4-t7l57\" (UID: \"c7dcae54-a627-4094-a940-3c5fb56d4bec\") " pod="kuadrant-system/authorino-6889cbd8c4-t7l57" Oct 30 00:52:26.166190 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:26.166122 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/839c0725-66a2-4100-bd25-da52516a001c-tls-cert\") pod \"authorino-5dfd97fb6f-4rk72\" (UID: \"839c0725-66a2-4100-bd25-da52516a001c\") " pod="kuadrant-system/authorino-5dfd97fb6f-4rk72" Oct 30 00:52:26.166190 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:26.166165 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzp8f\" (UniqueName: \"kubernetes.io/projected/839c0725-66a2-4100-bd25-da52516a001c-kube-api-access-wzp8f\") pod \"authorino-5dfd97fb6f-4rk72\" (UID: \"839c0725-66a2-4100-bd25-da52516a001c\") " pod="kuadrant-system/authorino-5dfd97fb6f-4rk72" Oct 30 00:52:26.168444 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:26.168418 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/839c0725-66a2-4100-bd25-da52516a001c-tls-cert\") pod \"authorino-5dfd97fb6f-4rk72\" (UID: \"839c0725-66a2-4100-bd25-da52516a001c\") " pod="kuadrant-system/authorino-5dfd97fb6f-4rk72" Oct 30 00:52:26.183052 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:26.183027 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-brzhj"] Oct 30 00:52:26.185832 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:26.185811 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzp8f\" (UniqueName: \"kubernetes.io/projected/839c0725-66a2-4100-bd25-da52516a001c-kube-api-access-wzp8f\") pod \"authorino-5dfd97fb6f-4rk72\" (UID: \"839c0725-66a2-4100-bd25-da52516a001c\") " pod="kuadrant-system/authorino-5dfd97fb6f-4rk72" Oct 30 00:52:26.187158 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:52:26.187136 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46878bd8_5b05_4220_9f14_dea40c76cc8e.slice/crio-40a510e91d537244205947d4273efa0e0c32b95fb7cb19b0accccaa12371c9de WatchSource:0}: Error finding container 40a510e91d537244205947d4273efa0e0c32b95fb7cb19b0accccaa12371c9de: Status 404 returned error can't find the container with id 40a510e91d537244205947d4273efa0e0c32b95fb7cb19b0accccaa12371c9de Oct 30 00:52:26.323917 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:26.323881 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dfd97fb6f-4rk72" Oct 30 00:52:26.637178 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:26.637142 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dfd97fb6f-4rk72"] Oct 30 00:52:26.641312 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:52:26.641289 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod839c0725_66a2_4100_bd25_da52516a001c.slice/crio-fc891d25ba5b79cc9253adb5180f6a0b2f74a7ed0cdefc2302a9ca2b756a30c3 WatchSource:0}: Error finding container fc891d25ba5b79cc9253adb5180f6a0b2f74a7ed0cdefc2302a9ca2b756a30c3: Status 404 returned error can't find the container with id fc891d25ba5b79cc9253adb5180f6a0b2f74a7ed0cdefc2302a9ca2b756a30c3 Oct 30 00:52:26.831785 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:26.831742 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-brzhj" event={"ID":"46878bd8-5b05-4220-9f14-dea40c76cc8e","Type":"ContainerStarted","Data":"40a510e91d537244205947d4273efa0e0c32b95fb7cb19b0accccaa12371c9de"} Oct 30 00:52:26.832829 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:26.832805 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dfd97fb6f-4rk72" event={"ID":"839c0725-66a2-4100-bd25-da52516a001c","Type":"ContainerStarted","Data":"fc891d25ba5b79cc9253adb5180f6a0b2f74a7ed0cdefc2302a9ca2b756a30c3"} Oct 30 00:52:26.833024 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:26.833008 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-t7l57" Oct 30 00:52:26.867851 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:26.865306 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-t7l57" Oct 30 00:52:26.971928 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:26.971902 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7dcae54-a627-4094-a940-3c5fb56d4bec-tls-cert\" (UniqueName: \"kubernetes.io/secret/c7dcae54-a627-4094-a940-3c5fb56d4bec-tls-cert\") pod \"c7dcae54-a627-4094-a940-3c5fb56d4bec\" (UID: \"c7dcae54-a627-4094-a940-3c5fb56d4bec\") " Oct 30 00:52:26.972119 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:26.971963 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7dcae54-a627-4094-a940-3c5fb56d4bec-kube-api-access-tmhbv\" (UniqueName: \"kubernetes.io/projected/c7dcae54-a627-4094-a940-3c5fb56d4bec-kube-api-access-tmhbv\") pod \"c7dcae54-a627-4094-a940-3c5fb56d4bec\" (UID: \"c7dcae54-a627-4094-a940-3c5fb56d4bec\") " Oct 30 00:52:26.974276 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:26.974246 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7dcae54-a627-4094-a940-3c5fb56d4bec-tls-cert" pod "c7dcae54-a627-4094-a940-3c5fb56d4bec" (UID: "c7dcae54-a627-4094-a940-3c5fb56d4bec"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:52:26.974276 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:26.974270 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7dcae54-a627-4094-a940-3c5fb56d4bec-kube-api-access-tmhbv" pod "c7dcae54-a627-4094-a940-3c5fb56d4bec" (UID: "c7dcae54-a627-4094-a940-3c5fb56d4bec"). InnerVolumeSpecName "kube-api-access-tmhbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:52:27.072534 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:27.072498 5300 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7dcae54-a627-4094-a940-3c5fb56d4bec-tls-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:52:27.072534 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:27.072528 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmhbv\" (UniqueName: \"kubernetes.io/projected/c7dcae54-a627-4094-a940-3c5fb56d4bec-kube-api-access-tmhbv\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:52:27.838091 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:27.838056 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-brzhj" event={"ID":"46878bd8-5b05-4220-9f14-dea40c76cc8e","Type":"ContainerStarted","Data":"fcdcbc2a50b1811b25b092ed1dbb87d62b112e8a74b1ced708855e01c1b4e267"} Oct 30 00:52:27.838343 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:27.838298 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-brzhj" podUID="46878bd8-5b05-4220-9f14-dea40c76cc8e" containerName="authorino" containerID="cri-o://fcdcbc2a50b1811b25b092ed1dbb87d62b112e8a74b1ced708855e01c1b4e267" gracePeriod=30 Oct 30 00:52:27.839616 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:27.839597 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-t7l57" Oct 30 00:52:27.839800 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:27.839632 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dfd97fb6f-4rk72" event={"ID":"839c0725-66a2-4100-bd25-da52516a001c","Type":"ContainerStarted","Data":"106ce3e5a96bfbe36b3b532cf08d7a7d45368ccf8ae30c47986f07d6c4bc75be"} Oct 30 00:52:27.874634 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:27.874592 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-brzhj" podStartSLOduration=2.283557277 podStartE2EDuration="2.874578278s" totalImagesPullingTime="591.021001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:27.87356622 +0000 UTC m=+548.189434542" watchObservedRunningTime="2026-10-30 00:52:27.874578278 +0000 UTC m=+548.190446602" Oct 30 00:52:27.930287 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:27.930249 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-t7l57"] Oct 30 00:52:27.931848 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:27.931817 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-t7l57"] Oct 30 00:52:27.966638 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:27.966585 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5dfd97fb6f-4rk72" podStartSLOduration=2.318519627 podStartE2EDuration="2.966569842s" totalImagesPullingTime="648.050215ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:27.966375504 +0000 UTC m=+548.282243841" watchObservedRunningTime="2026-10-30 00:52:27.966569842 +0000 UTC m=+548.282438167" Oct 30 00:52:27.990674 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:27.990637 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-pjfjd"] Oct 30 00:52:27.991529 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:27.990898 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-pjfjd" podUID="55d6773b-3df4-4a4a-bb3e-d895fdf8397c" containerName="authorino" containerID="cri-o://c4f757de61016fed68e3f2a38bfaa00d57cc93e7eb479e7872dadab7c8f81813" gracePeriod=30 Oct 30 00:52:28.122512 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:28.122383 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-brzhj" Oct 30 00:52:28.183647 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:28.183620 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46878bd8-5b05-4220-9f14-dea40c76cc8e-kube-api-access-fbdvg\" (UniqueName: \"kubernetes.io/projected/46878bd8-5b05-4220-9f14-dea40c76cc8e-kube-api-access-fbdvg\") pod \"46878bd8-5b05-4220-9f14-dea40c76cc8e\" (UID: \"46878bd8-5b05-4220-9f14-dea40c76cc8e\") " Oct 30 00:52:28.187736 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:28.187701 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46878bd8-5b05-4220-9f14-dea40c76cc8e-kube-api-access-fbdvg" pod "46878bd8-5b05-4220-9f14-dea40c76cc8e" (UID: "46878bd8-5b05-4220-9f14-dea40c76cc8e"). InnerVolumeSpecName "kube-api-access-fbdvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:52:28.242304 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:28.242280 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-pjfjd" Oct 30 00:52:28.284778 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:28.284737 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55d6773b-3df4-4a4a-bb3e-d895fdf8397c-kube-api-access-22jjg\" (UniqueName: \"kubernetes.io/projected/55d6773b-3df4-4a4a-bb3e-d895fdf8397c-kube-api-access-22jjg\") pod \"55d6773b-3df4-4a4a-bb3e-d895fdf8397c\" (UID: \"55d6773b-3df4-4a4a-bb3e-d895fdf8397c\") " Oct 30 00:52:28.285007 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:28.284990 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbdvg\" (UniqueName: \"kubernetes.io/projected/46878bd8-5b05-4220-9f14-dea40c76cc8e-kube-api-access-fbdvg\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:52:28.286911 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:28.286885 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d6773b-3df4-4a4a-bb3e-d895fdf8397c-kube-api-access-22jjg" pod "55d6773b-3df4-4a4a-bb3e-d895fdf8397c" (UID: "55d6773b-3df4-4a4a-bb3e-d895fdf8397c"). InnerVolumeSpecName "kube-api-access-22jjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:52:28.385497 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:28.385405 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22jjg\" (UniqueName: \"kubernetes.io/projected/55d6773b-3df4-4a4a-bb3e-d895fdf8397c-kube-api-access-22jjg\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:52:28.844181 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:28.844143 5300 generic.go:356] "Generic (PLEG): container finished" podID="55d6773b-3df4-4a4a-bb3e-d895fdf8397c" containerID="c4f757de61016fed68e3f2a38bfaa00d57cc93e7eb479e7872dadab7c8f81813" exitCode=0 Oct 30 00:52:28.844362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:28.844195 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-pjfjd" Oct 30 00:52:28.844362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:28.844228 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-pjfjd" event={"ID":"55d6773b-3df4-4a4a-bb3e-d895fdf8397c","Type":"ContainerDied","Data":"c4f757de61016fed68e3f2a38bfaa00d57cc93e7eb479e7872dadab7c8f81813"} Oct 30 00:52:28.844362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:28.844273 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-pjfjd" event={"ID":"55d6773b-3df4-4a4a-bb3e-d895fdf8397c","Type":"ContainerDied","Data":"ceac2ee00d69edacefff733d268f127d0673280e9ce139ea838a16f629ceca12"} Oct 30 00:52:28.844362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:28.844294 5300 scope.go:122] "RemoveContainer" containerID="c4f757de61016fed68e3f2a38bfaa00d57cc93e7eb479e7872dadab7c8f81813" Oct 30 00:52:28.845594 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:28.845394 5300 generic.go:356] "Generic (PLEG): container finished" podID="46878bd8-5b05-4220-9f14-dea40c76cc8e" containerID="fcdcbc2a50b1811b25b092ed1dbb87d62b112e8a74b1ced708855e01c1b4e267" exitCode=0 Oct 30 00:52:28.845594 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:28.845439 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-brzhj" Oct 30 00:52:28.845594 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:28.845470 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-brzhj" event={"ID":"46878bd8-5b05-4220-9f14-dea40c76cc8e","Type":"ContainerDied","Data":"fcdcbc2a50b1811b25b092ed1dbb87d62b112e8a74b1ced708855e01c1b4e267"} Oct 30 00:52:28.845594 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:28.845509 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-brzhj" event={"ID":"46878bd8-5b05-4220-9f14-dea40c76cc8e","Type":"ContainerDied","Data":"40a510e91d537244205947d4273efa0e0c32b95fb7cb19b0accccaa12371c9de"} Oct 30 00:52:28.852443 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:28.852426 5300 scope.go:122] "RemoveContainer" containerID="c4f757de61016fed68e3f2a38bfaa00d57cc93e7eb479e7872dadab7c8f81813" Oct 30 00:52:28.852714 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:52:28.852691 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f757de61016fed68e3f2a38bfaa00d57cc93e7eb479e7872dadab7c8f81813\": container with ID starting with c4f757de61016fed68e3f2a38bfaa00d57cc93e7eb479e7872dadab7c8f81813 not found: ID does not exist" containerID="c4f757de61016fed68e3f2a38bfaa00d57cc93e7eb479e7872dadab7c8f81813" Oct 30 00:52:28.852819 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:28.852724 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f757de61016fed68e3f2a38bfaa00d57cc93e7eb479e7872dadab7c8f81813"} err="failed to get container status \"c4f757de61016fed68e3f2a38bfaa00d57cc93e7eb479e7872dadab7c8f81813\": rpc error: code = NotFound desc = could not find container \"c4f757de61016fed68e3f2a38bfaa00d57cc93e7eb479e7872dadab7c8f81813\": container with ID starting with c4f757de61016fed68e3f2a38bfaa00d57cc93e7eb479e7872dadab7c8f81813 not found: ID does not exist" Oct 30 00:52:28.852819 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:28.852742 5300 scope.go:122] "RemoveContainer" containerID="fcdcbc2a50b1811b25b092ed1dbb87d62b112e8a74b1ced708855e01c1b4e267" Oct 30 00:52:28.860921 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:28.860902 5300 scope.go:122] "RemoveContainer" containerID="fcdcbc2a50b1811b25b092ed1dbb87d62b112e8a74b1ced708855e01c1b4e267" Oct 30 00:52:28.861154 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:52:28.861135 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcdcbc2a50b1811b25b092ed1dbb87d62b112e8a74b1ced708855e01c1b4e267\": container with ID starting with fcdcbc2a50b1811b25b092ed1dbb87d62b112e8a74b1ced708855e01c1b4e267 not found: ID does not exist" containerID="fcdcbc2a50b1811b25b092ed1dbb87d62b112e8a74b1ced708855e01c1b4e267" Oct 30 00:52:28.861201 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:28.861164 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcdcbc2a50b1811b25b092ed1dbb87d62b112e8a74b1ced708855e01c1b4e267"} err="failed to get container status \"fcdcbc2a50b1811b25b092ed1dbb87d62b112e8a74b1ced708855e01c1b4e267\": rpc error: code = NotFound desc = could not find container \"fcdcbc2a50b1811b25b092ed1dbb87d62b112e8a74b1ced708855e01c1b4e267\": container with ID starting with fcdcbc2a50b1811b25b092ed1dbb87d62b112e8a74b1ced708855e01c1b4e267 not found: ID does not exist" Oct 30 00:52:28.900468 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:28.900395 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-pjfjd"] Oct 30 00:52:28.904649 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:28.904624 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-pjfjd"] Oct 30 00:52:28.940853 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:28.940818 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-brzhj"] Oct 30 00:52:28.949307 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:28.949280 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-brzhj"] Oct 30 00:52:29.169549 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:29.169469 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:29.817913 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:29.817884 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46878bd8-5b05-4220-9f14-dea40c76cc8e" path="/var/lib/kubelet/pods/46878bd8-5b05-4220-9f14-dea40c76cc8e/volumes" Oct 30 00:52:29.818209 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:29.818196 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55d6773b-3df4-4a4a-bb3e-d895fdf8397c" path="/var/lib/kubelet/pods/55d6773b-3df4-4a4a-bb3e-d895fdf8397c/volumes" Oct 30 00:52:29.818501 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:29.818490 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7dcae54-a627-4094-a940-3c5fb56d4bec" path="/var/lib/kubelet/pods/c7dcae54-a627-4094-a940-3c5fb56d4bec/volumes" Oct 30 00:52:34.860185 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:34.860083 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:34.951632 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:34.951601 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-577f596567-fclnj"] Oct 30 00:52:34.951933 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:34.951918 5300 cpu_manager.go:414] "containerMap: removing container" podUID="46878bd8-5b05-4220-9f14-dea40c76cc8e" containerName="authorino" Oct 30 00:52:34.951933 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:34.951933 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46878bd8-5b05-4220-9f14-dea40c76cc8e" containerName="authorino" Oct 30 00:52:34.952058 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:34.951950 5300 cpu_manager.go:414] "containerMap: removing container" podUID="55d6773b-3df4-4a4a-bb3e-d895fdf8397c" containerName="authorino" Oct 30 00:52:34.952058 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:34.951955 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55d6773b-3df4-4a4a-bb3e-d895fdf8397c" containerName="authorino" Oct 30 00:52:34.952058 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:34.952022 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="46878bd8-5b05-4220-9f14-dea40c76cc8e" containerName="authorino" Oct 30 00:52:34.952058 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:34.952035 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="55d6773b-3df4-4a4a-bb3e-d895fdf8397c" containerName="authorino" Oct 30 00:52:34.954794 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:34.954773 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-fclnj" Oct 30 00:52:34.955153 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:34.955118 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:34.970199 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:34.970158 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 00:52:34.973021 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:34.973000 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 00:52:34.982295 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:34.982273 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 00:52:34.982400 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:34.982306 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-fmbv5\"" Oct 30 00:52:34.985992 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:34.985967 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-fclnj"] Oct 30 00:52:35.035100 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.035069 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:35.039192 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.039166 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4t6b\" (UniqueName: \"kubernetes.io/projected/19f0027e-c2cd-4274-8d80-4e5311bb4284-kube-api-access-t4t6b\") pod \"maas-controller-577f596567-fclnj\" (UID: \"19f0027e-c2cd-4274-8d80-4e5311bb4284\") " pod="opendatahub/maas-controller-577f596567-fclnj" Oct 30 00:52:35.039271 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.039224 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19f0027e-c2cd-4274-8d80-4e5311bb4284-webhook-cert\") pod \"maas-controller-577f596567-fclnj\" (UID: \"19f0027e-c2cd-4274-8d80-4e5311bb4284\") " pod="opendatahub/maas-controller-577f596567-fclnj" Oct 30 00:52:35.039345 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.039328 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19f0027e-c2cd-4274-8d80-4e5311bb4284-metrics-certs\") pod \"maas-controller-577f596567-fclnj\" (UID: \"19f0027e-c2cd-4274-8d80-4e5311bb4284\") " pod="opendatahub/maas-controller-577f596567-fclnj" Oct 30 00:52:35.140093 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.139996 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19f0027e-c2cd-4274-8d80-4e5311bb4284-webhook-cert\") pod \"maas-controller-577f596567-fclnj\" (UID: \"19f0027e-c2cd-4274-8d80-4e5311bb4284\") " pod="opendatahub/maas-controller-577f596567-fclnj" Oct 30 00:52:35.140093 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.140058 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19f0027e-c2cd-4274-8d80-4e5311bb4284-metrics-certs\") pod \"maas-controller-577f596567-fclnj\" (UID: \"19f0027e-c2cd-4274-8d80-4e5311bb4284\") " pod="opendatahub/maas-controller-577f596567-fclnj" Oct 30 00:52:35.140093 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.140103 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4t6b\" (UniqueName: \"kubernetes.io/projected/19f0027e-c2cd-4274-8d80-4e5311bb4284-kube-api-access-t4t6b\") pod \"maas-controller-577f596567-fclnj\" (UID: \"19f0027e-c2cd-4274-8d80-4e5311bb4284\") " pod="opendatahub/maas-controller-577f596567-fclnj" Oct 30 00:52:35.142523 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.142493 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19f0027e-c2cd-4274-8d80-4e5311bb4284-metrics-certs\") pod \"maas-controller-577f596567-fclnj\" (UID: \"19f0027e-c2cd-4274-8d80-4e5311bb4284\") " pod="opendatahub/maas-controller-577f596567-fclnj" Oct 30 00:52:35.142523 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.142513 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19f0027e-c2cd-4274-8d80-4e5311bb4284-webhook-cert\") pod \"maas-controller-577f596567-fclnj\" (UID: \"19f0027e-c2cd-4274-8d80-4e5311bb4284\") " pod="opendatahub/maas-controller-577f596567-fclnj" Oct 30 00:52:35.164837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.164805 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4t6b\" (UniqueName: \"kubernetes.io/projected/19f0027e-c2cd-4274-8d80-4e5311bb4284-kube-api-access-t4t6b\") pod \"maas-controller-577f596567-fclnj\" (UID: \"19f0027e-c2cd-4274-8d80-4e5311bb4284\") " pod="opendatahub/maas-controller-577f596567-fclnj" Oct 30 00:52:35.242982 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.242951 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:35.268765 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.268739 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-fclnj" Oct 30 00:52:35.325044 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.324719 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:35.329265 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.329233 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-59cfcff4f-mxd2h"] Oct 30 00:52:35.334042 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.334019 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59cfcff4f-mxd2h" Oct 30 00:52:35.352376 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.352321 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59cfcff4f-mxd2h"] Oct 30 00:52:35.443435 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.443347 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76bdf51b-ab36-40c2-9284-6cd0e71f1457-webhook-cert\") pod \"maas-controller-59cfcff4f-mxd2h\" (UID: \"76bdf51b-ab36-40c2-9284-6cd0e71f1457\") " pod="opendatahub/maas-controller-59cfcff4f-mxd2h" Oct 30 00:52:35.443435 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.443401 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76bdf51b-ab36-40c2-9284-6cd0e71f1457-metrics-certs\") pod \"maas-controller-59cfcff4f-mxd2h\" (UID: \"76bdf51b-ab36-40c2-9284-6cd0e71f1457\") " pod="opendatahub/maas-controller-59cfcff4f-mxd2h" Oct 30 00:52:35.443627 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.443523 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5f6c\" (UniqueName: \"kubernetes.io/projected/76bdf51b-ab36-40c2-9284-6cd0e71f1457-kube-api-access-m5f6c\") pod \"maas-controller-59cfcff4f-mxd2h\" (UID: \"76bdf51b-ab36-40c2-9284-6cd0e71f1457\") " pod="opendatahub/maas-controller-59cfcff4f-mxd2h" Oct 30 00:52:35.544746 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.544710 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5f6c\" (UniqueName: \"kubernetes.io/projected/76bdf51b-ab36-40c2-9284-6cd0e71f1457-kube-api-access-m5f6c\") pod \"maas-controller-59cfcff4f-mxd2h\" (UID: \"76bdf51b-ab36-40c2-9284-6cd0e71f1457\") " pod="opendatahub/maas-controller-59cfcff4f-mxd2h" Oct 30 00:52:35.544912 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.544865 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76bdf51b-ab36-40c2-9284-6cd0e71f1457-webhook-cert\") pod \"maas-controller-59cfcff4f-mxd2h\" (UID: \"76bdf51b-ab36-40c2-9284-6cd0e71f1457\") " pod="opendatahub/maas-controller-59cfcff4f-mxd2h" Oct 30 00:52:35.544912 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.544894 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76bdf51b-ab36-40c2-9284-6cd0e71f1457-metrics-certs\") pod \"maas-controller-59cfcff4f-mxd2h\" (UID: \"76bdf51b-ab36-40c2-9284-6cd0e71f1457\") " pod="opendatahub/maas-controller-59cfcff4f-mxd2h" Oct 30 00:52:35.547315 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.547292 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76bdf51b-ab36-40c2-9284-6cd0e71f1457-metrics-certs\") pod \"maas-controller-59cfcff4f-mxd2h\" (UID: \"76bdf51b-ab36-40c2-9284-6cd0e71f1457\") " pod="opendatahub/maas-controller-59cfcff4f-mxd2h" Oct 30 00:52:35.547424 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.547339 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76bdf51b-ab36-40c2-9284-6cd0e71f1457-webhook-cert\") pod \"maas-controller-59cfcff4f-mxd2h\" (UID: \"76bdf51b-ab36-40c2-9284-6cd0e71f1457\") " pod="opendatahub/maas-controller-59cfcff4f-mxd2h" Oct 30 00:52:35.560374 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.560346 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-fclnj"] Oct 30 00:52:35.563923 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.563900 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5f6c\" (UniqueName: \"kubernetes.io/projected/76bdf51b-ab36-40c2-9284-6cd0e71f1457-kube-api-access-m5f6c\") pod \"maas-controller-59cfcff4f-mxd2h\" (UID: \"76bdf51b-ab36-40c2-9284-6cd0e71f1457\") " pod="opendatahub/maas-controller-59cfcff4f-mxd2h" Oct 30 00:52:35.564826 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:52:35.564802 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f0027e_c2cd_4274_8d80_4e5311bb4284.slice/crio-17fcded0c2d2f02aaadc71ef4db06f07b8ee8d264df01b9aaa202be2ce8270d1 WatchSource:0}: Error finding container 17fcded0c2d2f02aaadc71ef4db06f07b8ee8d264df01b9aaa202be2ce8270d1: Status 404 returned error can't find the container with id 17fcded0c2d2f02aaadc71ef4db06f07b8ee8d264df01b9aaa202be2ce8270d1 Oct 30 00:52:35.647347 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.647307 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59cfcff4f-mxd2h" Oct 30 00:52:35.875654 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.875620 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-fclnj" event={"ID":"19f0027e-c2cd-4274-8d80-4e5311bb4284","Type":"ContainerStarted","Data":"17fcded0c2d2f02aaadc71ef4db06f07b8ee8d264df01b9aaa202be2ce8270d1"} Oct 30 00:52:35.936737 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:35.936698 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59cfcff4f-mxd2h"] Oct 30 00:52:35.941008 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:52:35.940974 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bdf51b_ab36_40c2_9284_6cd0e71f1457.slice/crio-1af067d3720eaa15aeae1116fefa7b542f4d51e121383bc72e90eb83b9ccbaf6 WatchSource:0}: Error finding container 1af067d3720eaa15aeae1116fefa7b542f4d51e121383bc72e90eb83b9ccbaf6: Status 404 returned error can't find the container with id 1af067d3720eaa15aeae1116fefa7b542f4d51e121383bc72e90eb83b9ccbaf6 Oct 30 00:52:36.884433 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:36.884347 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59cfcff4f-mxd2h" event={"ID":"76bdf51b-ab36-40c2-9284-6cd0e71f1457","Type":"ContainerStarted","Data":"1af067d3720eaa15aeae1116fefa7b542f4d51e121383bc72e90eb83b9ccbaf6"} Oct 30 00:52:38.825447 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:38.825361 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:38.895671 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:38.895633 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59cfcff4f-mxd2h" event={"ID":"76bdf51b-ab36-40c2-9284-6cd0e71f1457","Type":"ContainerStarted","Data":"1ccea0c94c41cb78b0a0c99a6cf719917fe14a5ac9094ba0b7fd5bdbb71380d7"} Oct 30 00:52:38.895886 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:38.895796 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-59cfcff4f-mxd2h" Oct 30 00:52:38.897039 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:38.897017 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-fclnj" event={"ID":"19f0027e-c2cd-4274-8d80-4e5311bb4284","Type":"ContainerStarted","Data":"c8283a8328d05498e5fb7488191d16899e4ea096181ee06ad060c626e7a23a5d"} Oct 30 00:52:38.897175 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:38.897069 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-577f596567-fclnj" Oct 30 00:52:38.925662 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:38.925634 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:38.942615 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:38.942577 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-59cfcff4f-mxd2h" podStartSLOduration=1.363025996 podStartE2EDuration="3.942563923s" totalImagesPullingTime="2.579537927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:38.941244682 +0000 UTC m=+559.257113007" watchObservedRunningTime="2026-10-30 00:52:38.942563923 +0000 UTC m=+559.258432279" Oct 30 00:52:38.982733 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:38.982666 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-577f596567-fclnj" podStartSLOduration=2.250746515 podStartE2EDuration="4.98264627s" totalImagesPullingTime="2.731899755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:38.980500007 +0000 UTC m=+559.296368332" watchObservedRunningTime="2026-10-30 00:52:38.98264627 +0000 UTC m=+559.298514599" Oct 30 00:52:42.160583 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:42.160537 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-997pg"] Oct 30 00:52:42.164209 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:42.164180 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-997pg" Oct 30 00:52:42.178703 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:42.178675 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 00:52:42.178703 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:42.178691 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 00:52:42.178907 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:42.178746 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-vq284\"" Oct 30 00:52:42.190888 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:42.190861 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 00:52:42.191926 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:42.191902 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-997pg"] Oct 30 00:52:42.232105 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:42.232075 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:42.308503 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:42.308466 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/917950da-2187-4df2-972b-89c4dc58a547-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-997pg\" (UID: \"917950da-2187-4df2-972b-89c4dc58a547\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-997pg" Oct 30 00:52:42.308503 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:42.308503 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/917950da-2187-4df2-972b-89c4dc58a547-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-997pg\" (UID: \"917950da-2187-4df2-972b-89c4dc58a547\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-997pg" Oct 30 00:52:42.308728 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:42.308530 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22d2c\" (UniqueName: \"kubernetes.io/projected/917950da-2187-4df2-972b-89c4dc58a547-kube-api-access-22d2c\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-997pg\" (UID: \"917950da-2187-4df2-972b-89c4dc58a547\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-997pg" Oct 30 00:52:42.308728 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:42.308671 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/917950da-2187-4df2-972b-89c4dc58a547-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-997pg\" (UID: \"917950da-2187-4df2-972b-89c4dc58a547\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-997pg" Oct 30 00:52:42.332390 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:42.332363 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:52:42.409884 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:42.409847 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/917950da-2187-4df2-972b-89c4dc58a547-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-997pg\" (UID: \"917950da-2187-4df2-972b-89c4dc58a547\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-997pg" Oct 30 00:52:42.410061 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:42.409941 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/917950da-2187-4df2-972b-89c4dc58a547-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-997pg\" (UID: \"917950da-2187-4df2-972b-89c4dc58a547\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-997pg" Oct 30 00:52:42.410061 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:42.409966 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/917950da-2187-4df2-972b-89c4dc58a547-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-997pg\" (UID: \"917950da-2187-4df2-972b-89c4dc58a547\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-997pg" Oct 30 00:52:42.410061 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:42.409991 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22d2c\" (UniqueName: \"kubernetes.io/projected/917950da-2187-4df2-972b-89c4dc58a547-kube-api-access-22d2c\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-997pg\" (UID: \"917950da-2187-4df2-972b-89c4dc58a547\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-997pg" Oct 30 00:52:42.410499 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:42.410476 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/917950da-2187-4df2-972b-89c4dc58a547-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-997pg\" (UID: \"917950da-2187-4df2-972b-89c4dc58a547\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-997pg" Oct 30 00:52:42.410698 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:42.410649 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/917950da-2187-4df2-972b-89c4dc58a547-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-997pg\" (UID: \"917950da-2187-4df2-972b-89c4dc58a547\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-997pg" Oct 30 00:52:42.412350 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:42.412330 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/917950da-2187-4df2-972b-89c4dc58a547-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-997pg\" (UID: \"917950da-2187-4df2-972b-89c4dc58a547\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-997pg" Oct 30 00:52:42.429151 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:42.429118 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22d2c\" (UniqueName: \"kubernetes.io/projected/917950da-2187-4df2-972b-89c4dc58a547-kube-api-access-22d2c\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-997pg\" (UID: \"917950da-2187-4df2-972b-89c4dc58a547\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-997pg" Oct 30 00:52:42.476571 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:42.476535 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-997pg" Oct 30 00:52:42.766849 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:42.766813 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-997pg"] Oct 30 00:52:42.770712 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:52:42.770686 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod917950da_2187_4df2_972b_89c4dc58a547.slice/crio-624cc2668713e4073a220fa3ff3130521ddeb3c3e6b8240f3808b550d6841b4e WatchSource:0}: Error finding container 624cc2668713e4073a220fa3ff3130521ddeb3c3e6b8240f3808b550d6841b4e: Status 404 returned error can't find the container with id 624cc2668713e4073a220fa3ff3130521ddeb3c3e6b8240f3808b550d6841b4e Oct 30 00:52:42.912943 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:42.912900 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-997pg" event={"ID":"917950da-2187-4df2-972b-89c4dc58a547","Type":"ContainerStarted","Data":"624cc2668713e4073a220fa3ff3130521ddeb3c3e6b8240f3808b550d6841b4e"} Oct 30 00:52:49.903277 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:49.903243 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-59cfcff4f-mxd2h" Oct 30 00:52:49.903892 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:49.903322 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-577f596567-fclnj" Oct 30 00:52:49.969801 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:49.969382 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-fclnj"] Oct 30 00:52:49.969801 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:49.969614 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-577f596567-fclnj" podUID="19f0027e-c2cd-4274-8d80-4e5311bb4284" containerName="manager" containerID="cri-o://c8283a8328d05498e5fb7488191d16899e4ea096181ee06ad060c626e7a23a5d" gracePeriod=30 Oct 30 00:52:50.243346 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:50.243314 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-fclnj" Oct 30 00:52:50.384703 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:50.384654 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19f0027e-c2cd-4274-8d80-4e5311bb4284-kube-api-access-t4t6b\" (UniqueName: \"kubernetes.io/projected/19f0027e-c2cd-4274-8d80-4e5311bb4284-kube-api-access-t4t6b\") pod \"19f0027e-c2cd-4274-8d80-4e5311bb4284\" (UID: \"19f0027e-c2cd-4274-8d80-4e5311bb4284\") " Oct 30 00:52:50.384922 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:50.384721 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19f0027e-c2cd-4274-8d80-4e5311bb4284-webhook-cert\" (UniqueName: \"kubernetes.io/secret/19f0027e-c2cd-4274-8d80-4e5311bb4284-webhook-cert\") pod \"19f0027e-c2cd-4274-8d80-4e5311bb4284\" (UID: \"19f0027e-c2cd-4274-8d80-4e5311bb4284\") " Oct 30 00:52:50.384922 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:50.384838 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19f0027e-c2cd-4274-8d80-4e5311bb4284-metrics-certs\" (UniqueName: \"kubernetes.io/secret/19f0027e-c2cd-4274-8d80-4e5311bb4284-metrics-certs\") pod \"19f0027e-c2cd-4274-8d80-4e5311bb4284\" (UID: \"19f0027e-c2cd-4274-8d80-4e5311bb4284\") " Oct 30 00:52:50.387513 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:50.387471 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19f0027e-c2cd-4274-8d80-4e5311bb4284-metrics-certs" pod "19f0027e-c2cd-4274-8d80-4e5311bb4284" (UID: "19f0027e-c2cd-4274-8d80-4e5311bb4284"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:52:50.387513 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:50.387469 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f0027e-c2cd-4274-8d80-4e5311bb4284-kube-api-access-t4t6b" pod "19f0027e-c2cd-4274-8d80-4e5311bb4284" (UID: "19f0027e-c2cd-4274-8d80-4e5311bb4284"). InnerVolumeSpecName "kube-api-access-t4t6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:52:50.387735 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:50.387566 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19f0027e-c2cd-4274-8d80-4e5311bb4284-webhook-cert" pod "19f0027e-c2cd-4274-8d80-4e5311bb4284" (UID: "19f0027e-c2cd-4274-8d80-4e5311bb4284"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:52:50.486409 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:50.486369 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4t6b\" (UniqueName: \"kubernetes.io/projected/19f0027e-c2cd-4274-8d80-4e5311bb4284-kube-api-access-t4t6b\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:52:50.486409 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:50.486414 5300 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19f0027e-c2cd-4274-8d80-4e5311bb4284-webhook-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:52:50.486636 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:50.486431 5300 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19f0027e-c2cd-4274-8d80-4e5311bb4284-metrics-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:52:50.954098 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:50.954004 5300 generic.go:356] "Generic (PLEG): container finished" podID="19f0027e-c2cd-4274-8d80-4e5311bb4284" containerID="c8283a8328d05498e5fb7488191d16899e4ea096181ee06ad060c626e7a23a5d" exitCode=0 Oct 30 00:52:50.954098 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:50.954062 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-fclnj" event={"ID":"19f0027e-c2cd-4274-8d80-4e5311bb4284","Type":"ContainerDied","Data":"c8283a8328d05498e5fb7488191d16899e4ea096181ee06ad060c626e7a23a5d"} Oct 30 00:52:50.954610 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:50.954110 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-fclnj" event={"ID":"19f0027e-c2cd-4274-8d80-4e5311bb4284","Type":"ContainerDied","Data":"17fcded0c2d2f02aaadc71ef4db06f07b8ee8d264df01b9aaa202be2ce8270d1"} Oct 30 00:52:50.954610 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:50.954136 5300 scope.go:122] "RemoveContainer" containerID="c8283a8328d05498e5fb7488191d16899e4ea096181ee06ad060c626e7a23a5d" Oct 30 00:52:50.954610 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:50.954061 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-fclnj" Oct 30 00:52:50.963916 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:50.963895 5300 scope.go:122] "RemoveContainer" containerID="c8283a8328d05498e5fb7488191d16899e4ea096181ee06ad060c626e7a23a5d" Oct 30 00:52:50.964275 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:52:50.964246 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8283a8328d05498e5fb7488191d16899e4ea096181ee06ad060c626e7a23a5d\": container with ID starting with c8283a8328d05498e5fb7488191d16899e4ea096181ee06ad060c626e7a23a5d not found: ID does not exist" containerID="c8283a8328d05498e5fb7488191d16899e4ea096181ee06ad060c626e7a23a5d" Oct 30 00:52:50.964354 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:50.964288 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8283a8328d05498e5fb7488191d16899e4ea096181ee06ad060c626e7a23a5d"} err="failed to get container status \"c8283a8328d05498e5fb7488191d16899e4ea096181ee06ad060c626e7a23a5d\": rpc error: code = NotFound desc = could not find container \"c8283a8328d05498e5fb7488191d16899e4ea096181ee06ad060c626e7a23a5d\": container with ID starting with c8283a8328d05498e5fb7488191d16899e4ea096181ee06ad060c626e7a23a5d not found: ID does not exist" Oct 30 00:52:51.003670 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:51.003634 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-fclnj"] Oct 30 00:52:51.015501 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:51.015319 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-577f596567-fclnj"] Oct 30 00:52:51.597696 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:52:51.597653 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:52:51.818969 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:51.818933 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19f0027e-c2cd-4274-8d80-4e5311bb4284" path="/var/lib/kubelet/pods/19f0027e-c2cd-4274-8d80-4e5311bb4284/volumes" Oct 30 00:52:54.973679 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:54.973642 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-997pg" event={"ID":"917950da-2187-4df2-972b-89c4dc58a547","Type":"ContainerStarted","Data":"09f0b11cd351a269126e18f8a7d41a1b09858dd4309296518862b1f640325d7a"} Oct 30 00:52:54.974226 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:54.973781 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-997pg" Oct 30 00:52:55.011568 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:52:55.011521 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-997pg" podStartSLOduration=1.347255191 podStartE2EDuration="13.01150729s" totalImagesPullingTime="11.664252099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:55.010676081 +0000 UTC m=+575.326544408" watchObservedRunningTime="2026-10-30 00:52:55.01150729 +0000 UTC m=+575.327375615" Oct 30 00:53:05.983203 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:05.983170 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-997pg" Oct 30 00:53:10.955731 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:10.955696 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:11.087677 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:11.087627 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:12.502929 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:12.502898 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:12.603467 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:12.603426 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:12.714120 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:12.714089 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:12.758363 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:12.758285 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln"] Oct 30 00:53:12.758699 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:12.758674 5300 cpu_manager.go:414] "containerMap: removing container" podUID="19f0027e-c2cd-4274-8d80-4e5311bb4284" containerName="manager" Oct 30 00:53:12.758699 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:12.758696 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19f0027e-c2cd-4274-8d80-4e5311bb4284" containerName="manager" Oct 30 00:53:12.758866 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:12.758819 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="19f0027e-c2cd-4274-8d80-4e5311bb4284" containerName="manager" Oct 30 00:53:12.811774 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:12.811724 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:13.034400 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:13.034319 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln"] Oct 30 00:53:13.034542 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:13.034445 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln" Oct 30 00:53:13.060842 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:13.060815 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 00:53:13.060842 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:13.060833 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nss4l\"" Oct 30 00:53:13.061081 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:13.060902 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 00:53:13.072973 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:13.072940 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c9985b-f3ab-4e84-8db9-41a954d358a6-maas-api-tls\") pod \"maas-api-5c48877c75-4k9ln\" (UID: \"57c9985b-f3ab-4e84-8db9-41a954d358a6\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln" Oct 30 00:53:13.072973 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:13.072978 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c9985b-f3ab-4e84-8db9-41a954d358a6-openshift-service-ca\") pod \"maas-api-5c48877c75-4k9ln\" (UID: \"57c9985b-f3ab-4e84-8db9-41a954d358a6\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln" Oct 30 00:53:13.073200 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:13.073014 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c9985b-f3ab-4e84-8db9-41a954d358a6-metrics-certs\") pod \"maas-api-5c48877c75-4k9ln\" (UID: \"57c9985b-f3ab-4e84-8db9-41a954d358a6\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln" Oct 30 00:53:13.073200 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:13.073080 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg9rm\" (UniqueName: \"kubernetes.io/projected/57c9985b-f3ab-4e84-8db9-41a954d358a6-kube-api-access-wg9rm\") pod \"maas-api-5c48877c75-4k9ln\" (UID: \"57c9985b-f3ab-4e84-8db9-41a954d358a6\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln" Oct 30 00:53:13.174049 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:13.174013 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c9985b-f3ab-4e84-8db9-41a954d358a6-metrics-certs\") pod \"maas-api-5c48877c75-4k9ln\" (UID: \"57c9985b-f3ab-4e84-8db9-41a954d358a6\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln" Oct 30 00:53:13.174254 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:13.174063 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg9rm\" (UniqueName: \"kubernetes.io/projected/57c9985b-f3ab-4e84-8db9-41a954d358a6-kube-api-access-wg9rm\") pod \"maas-api-5c48877c75-4k9ln\" (UID: \"57c9985b-f3ab-4e84-8db9-41a954d358a6\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln" Oct 30 00:53:13.174254 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:13.174121 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c9985b-f3ab-4e84-8db9-41a954d358a6-maas-api-tls\") pod \"maas-api-5c48877c75-4k9ln\" (UID: \"57c9985b-f3ab-4e84-8db9-41a954d358a6\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln" Oct 30 00:53:13.174254 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:13.174161 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c9985b-f3ab-4e84-8db9-41a954d358a6-openshift-service-ca\") pod \"maas-api-5c48877c75-4k9ln\" (UID: \"57c9985b-f3ab-4e84-8db9-41a954d358a6\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln" Oct 30 00:53:13.174813 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:13.174789 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c9985b-f3ab-4e84-8db9-41a954d358a6-openshift-service-ca\") pod \"maas-api-5c48877c75-4k9ln\" (UID: \"57c9985b-f3ab-4e84-8db9-41a954d358a6\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln" Oct 30 00:53:13.176465 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:13.176445 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c9985b-f3ab-4e84-8db9-41a954d358a6-metrics-certs\") pod \"maas-api-5c48877c75-4k9ln\" (UID: \"57c9985b-f3ab-4e84-8db9-41a954d358a6\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln" Oct 30 00:53:13.176603 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:13.176583 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c9985b-f3ab-4e84-8db9-41a954d358a6-maas-api-tls\") pod \"maas-api-5c48877c75-4k9ln\" (UID: \"57c9985b-f3ab-4e84-8db9-41a954d358a6\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln" Oct 30 00:53:13.193113 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:13.193086 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg9rm\" (UniqueName: \"kubernetes.io/projected/57c9985b-f3ab-4e84-8db9-41a954d358a6-kube-api-access-wg9rm\") pod \"maas-api-5c48877c75-4k9ln\" (UID: \"57c9985b-f3ab-4e84-8db9-41a954d358a6\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln" Oct 30 00:53:13.365030 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:13.364958 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln" Oct 30 00:53:13.675118 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:13.675034 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln"] Oct 30 00:53:13.678889 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:53:13.678864 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c9985b_f3ab_4e84_8db9_41a954d358a6.slice/crio-8868191ba27b5601bd487e322289d347181a6c058779c357c7eafa0f48cd2df1 WatchSource:0}: Error finding container 8868191ba27b5601bd487e322289d347181a6c058779c357c7eafa0f48cd2df1: Status 404 returned error can't find the container with id 8868191ba27b5601bd487e322289d347181a6c058779c357c7eafa0f48cd2df1 Oct 30 00:53:14.049335 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:14.049303 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln" event={"ID":"57c9985b-f3ab-4e84-8db9-41a954d358a6","Type":"ContainerStarted","Data":"8868191ba27b5601bd487e322289d347181a6c058779c357c7eafa0f48cd2df1"} Oct 30 00:53:16.059231 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:16.059198 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln" event={"ID":"57c9985b-f3ab-4e84-8db9-41a954d358a6","Type":"ContainerStarted","Data":"e507abfe333c7bc21bbf1009b178bc6b037a51de612e94e8ddf38afcf456b8e7"} Oct 30 00:53:16.059834 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:16.059260 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln" Oct 30 00:53:16.096780 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:16.096717 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln" podStartSLOduration=2.243527145 podStartE2EDuration="4.096703347s" totalImagesPullingTime="1.853176202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:16.096585584 +0000 UTC m=+596.412453910" watchObservedRunningTime="2026-10-30 00:53:16.096703347 +0000 UTC m=+596.412571672" Oct 30 00:53:21.591584 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:53:21.591543 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:53:22.068866 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:22.068834 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln" Oct 30 00:53:29.304129 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:29.304092 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:48.397745 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:48.397708 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:51.591594 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:53:51.591550 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:53:56.988994 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:56.988963 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:53:57.094359 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:53:57.094307 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:15.401016 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:15.400919 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:16.967283 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:16.967246 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh"] Oct 30 00:54:16.970853 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:16.970826 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh" Oct 30 00:54:16.986582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:16.986554 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 00:54:16.998530 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:16.998503 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 00:54:16.998530 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:16.998520 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 00:54:16.998747 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:16.998560 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xc4jj\"" Oct 30 00:54:16.999710 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:16.999693 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh"] Oct 30 00:54:17.024943 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:17.024911 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-696nh\" (UID: \"ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh" Oct 30 00:54:17.024943 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:17.024945 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwbth\" (UniqueName: \"kubernetes.io/projected/ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720-kube-api-access-lwbth\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-696nh\" (UID: \"ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh" Oct 30 00:54:17.025129 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:17.025023 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-696nh\" (UID: \"ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh" Oct 30 00:54:17.025129 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:17.025040 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-696nh\" (UID: \"ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh" Oct 30 00:54:17.025129 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:17.025113 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-696nh\" (UID: \"ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh" Oct 30 00:54:17.025223 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:17.025181 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-696nh\" (UID: \"ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh" Oct 30 00:54:17.037545 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:17.037518 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:17.126254 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:17.126205 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-696nh\" (UID: \"ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh" Oct 30 00:54:17.126454 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:17.126272 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-696nh\" (UID: \"ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh" Oct 30 00:54:17.126454 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:17.126301 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwbth\" (UniqueName: \"kubernetes.io/projected/ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720-kube-api-access-lwbth\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-696nh\" (UID: \"ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh" Oct 30 00:54:17.126454 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:17.126358 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-696nh\" (UID: \"ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh" Oct 30 00:54:17.126454 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:17.126384 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-696nh\" (UID: \"ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh" Oct 30 00:54:17.126454 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:17.126417 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-696nh\" (UID: \"ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh" Oct 30 00:54:17.126799 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:17.126724 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-696nh\" (UID: \"ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh" Oct 30 00:54:17.126799 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:17.126756 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-696nh\" (UID: \"ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh" Oct 30 00:54:17.127555 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:17.127255 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-696nh\" (UID: \"ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh" Oct 30 00:54:17.128729 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:17.128703 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-696nh\" (UID: \"ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh" Oct 30 00:54:17.130133 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:17.130109 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-696nh\" (UID: \"ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh" Oct 30 00:54:17.145911 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:17.145881 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwbth\" (UniqueName: \"kubernetes.io/projected/ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720-kube-api-access-lwbth\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-696nh\" (UID: \"ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh" Oct 30 00:54:17.283885 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:17.283838 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh" Oct 30 00:54:17.579936 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:17.579834 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh"] Oct 30 00:54:17.583666 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:54:17.583636 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac36d5c9_9a67_4ca1_abc6_f47ff1bcf720.slice/crio-8d1b5c75dc1d70ae7fb7a5de9edd04a4b6ccf62cdc58f58539146f37c9fc0894 WatchSource:0}: Error finding container 8d1b5c75dc1d70ae7fb7a5de9edd04a4b6ccf62cdc58f58539146f37c9fc0894: Status 404 returned error can't find the container with id 8d1b5c75dc1d70ae7fb7a5de9edd04a4b6ccf62cdc58f58539146f37c9fc0894 Oct 30 00:54:18.100011 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:18.099971 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:18.313954 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:18.313921 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh" event={"ID":"ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720","Type":"ContainerStarted","Data":"8d1b5c75dc1d70ae7fb7a5de9edd04a4b6ccf62cdc58f58539146f37c9fc0894"} Oct 30 00:54:18.365548 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:18.365448 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd"] Oct 30 00:54:18.377023 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:18.376997 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd" Oct 30 00:54:18.390870 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:18.390838 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd"] Oct 30 00:54:18.401882 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:18.401858 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 00:54:18.416655 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:18.416624 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:18.439948 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:18.439912 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b27f00f-ab83-4393-9223-8b4e790dfa28-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kzbhd\" (UID: \"1b27f00f-ab83-4393-9223-8b4e790dfa28\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd" Oct 30 00:54:18.440093 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:18.439960 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b27f00f-ab83-4393-9223-8b4e790dfa28-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kzbhd\" (UID: \"1b27f00f-ab83-4393-9223-8b4e790dfa28\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd" Oct 30 00:54:18.440148 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:18.440086 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b27f00f-ab83-4393-9223-8b4e790dfa28-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kzbhd\" (UID: \"1b27f00f-ab83-4393-9223-8b4e790dfa28\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd" Oct 30 00:54:18.440148 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:18.440114 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-795p7\" (UniqueName: \"kubernetes.io/projected/1b27f00f-ab83-4393-9223-8b4e790dfa28-kube-api-access-795p7\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kzbhd\" (UID: \"1b27f00f-ab83-4393-9223-8b4e790dfa28\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd" Oct 30 00:54:18.440247 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:18.440222 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b27f00f-ab83-4393-9223-8b4e790dfa28-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kzbhd\" (UID: \"1b27f00f-ab83-4393-9223-8b4e790dfa28\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd" Oct 30 00:54:18.440285 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:18.440270 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b27f00f-ab83-4393-9223-8b4e790dfa28-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kzbhd\" (UID: \"1b27f00f-ab83-4393-9223-8b4e790dfa28\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd" Oct 30 00:54:18.541433 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:18.541395 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b27f00f-ab83-4393-9223-8b4e790dfa28-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kzbhd\" (UID: \"1b27f00f-ab83-4393-9223-8b4e790dfa28\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd" Oct 30 00:54:18.541615 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:18.541447 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b27f00f-ab83-4393-9223-8b4e790dfa28-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kzbhd\" (UID: \"1b27f00f-ab83-4393-9223-8b4e790dfa28\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd" Oct 30 00:54:18.541615 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:18.541488 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b27f00f-ab83-4393-9223-8b4e790dfa28-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kzbhd\" (UID: \"1b27f00f-ab83-4393-9223-8b4e790dfa28\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd" Oct 30 00:54:18.541615 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:18.541513 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-795p7\" (UniqueName: \"kubernetes.io/projected/1b27f00f-ab83-4393-9223-8b4e790dfa28-kube-api-access-795p7\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kzbhd\" (UID: \"1b27f00f-ab83-4393-9223-8b4e790dfa28\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd" Oct 30 00:54:18.541615 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:18.541557 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b27f00f-ab83-4393-9223-8b4e790dfa28-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kzbhd\" (UID: \"1b27f00f-ab83-4393-9223-8b4e790dfa28\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd" Oct 30 00:54:18.541615 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:18.541589 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b27f00f-ab83-4393-9223-8b4e790dfa28-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kzbhd\" (UID: \"1b27f00f-ab83-4393-9223-8b4e790dfa28\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd" Oct 30 00:54:18.542054 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:18.542032 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b27f00f-ab83-4393-9223-8b4e790dfa28-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kzbhd\" (UID: \"1b27f00f-ab83-4393-9223-8b4e790dfa28\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd" Oct 30 00:54:18.542214 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:18.542186 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b27f00f-ab83-4393-9223-8b4e790dfa28-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kzbhd\" (UID: \"1b27f00f-ab83-4393-9223-8b4e790dfa28\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd" Oct 30 00:54:18.542501 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:18.542479 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b27f00f-ab83-4393-9223-8b4e790dfa28-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kzbhd\" (UID: \"1b27f00f-ab83-4393-9223-8b4e790dfa28\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd" Oct 30 00:54:18.544674 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:18.544646 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b27f00f-ab83-4393-9223-8b4e790dfa28-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kzbhd\" (UID: \"1b27f00f-ab83-4393-9223-8b4e790dfa28\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd" Oct 30 00:54:18.545115 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:18.545082 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b27f00f-ab83-4393-9223-8b4e790dfa28-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kzbhd\" (UID: \"1b27f00f-ab83-4393-9223-8b4e790dfa28\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd" Oct 30 00:54:18.560452 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:18.560423 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-795p7\" (UniqueName: \"kubernetes.io/projected/1b27f00f-ab83-4393-9223-8b4e790dfa28-kube-api-access-795p7\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kzbhd\" (UID: \"1b27f00f-ab83-4393-9223-8b4e790dfa28\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd" Oct 30 00:54:18.704196 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:18.703935 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd" Oct 30 00:54:19.006700 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:19.006651 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd"] Oct 30 00:54:19.011967 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:54:19.011937 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b27f00f_ab83_4393_9223_8b4e790dfa28.slice/crio-d6f4e6d8db144d52d775beaf09550fb2405fece86cdb36c59e453f8ae2165ab2 WatchSource:0}: Error finding container d6f4e6d8db144d52d775beaf09550fb2405fece86cdb36c59e453f8ae2165ab2: Status 404 returned error can't find the container with id d6f4e6d8db144d52d775beaf09550fb2405fece86cdb36c59e453f8ae2165ab2 Oct 30 00:54:19.320113 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:19.320018 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd" event={"ID":"1b27f00f-ab83-4393-9223-8b4e790dfa28","Type":"ContainerStarted","Data":"43fc795836442dbcf8aeb26b8627f72fb18aa3d70553c1ca583c34c7eb22fac6"} Oct 30 00:54:19.320113 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:19.320081 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd" event={"ID":"1b27f00f-ab83-4393-9223-8b4e790dfa28","Type":"ContainerStarted","Data":"d6f4e6d8db144d52d775beaf09550fb2405fece86cdb36c59e453f8ae2165ab2"} Oct 30 00:54:19.321858 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:19.321834 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh" event={"ID":"ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720","Type":"ContainerStarted","Data":"2f0fffc2338b42d4c474213b1fd8eb690efe6c4cdbd288bae7f4a93f7890cf50"} Oct 30 00:54:19.360421 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:19.360354 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd" podStartSLOduration=1.179968174 podStartE2EDuration="1.360333272s" totalImagesPullingTime="180.365098ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:19.359316595 +0000 UTC m=+659.675184914" watchObservedRunningTime="2026-10-30 00:54:19.360333272 +0000 UTC m=+659.676201598" Oct 30 00:54:19.397466 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:19.397409 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh" podStartSLOduration=2.361215028 podStartE2EDuration="3.397394782s" totalImagesPullingTime="1.036179754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:19.397082009 +0000 UTC m=+659.712950334" watchObservedRunningTime="2026-10-30 00:54:19.397394782 +0000 UTC m=+659.713263108" Oct 30 00:54:19.599120 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:19.599026 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:19.704399 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:19.704349 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd" Oct 30 00:54:20.284677 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:20.284643 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh" Oct 30 00:54:21.593710 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:54:21.593672 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:54:23.584404 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:23.584363 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t"] Oct 30 00:54:23.588104 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:23.588081 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t" Oct 30 00:54:23.605811 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:23.605779 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t"] Oct 30 00:54:23.616774 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:23.616732 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 00:54:23.646477 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:23.646449 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:23.724952 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:23.724918 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dc03dfd-8c9e-42a1-9677-361219162734-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-k659t\" (UID: \"4dc03dfd-8c9e-42a1-9677-361219162734\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t" Oct 30 00:54:23.724952 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:23.724953 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dc03dfd-8c9e-42a1-9677-361219162734-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-k659t\" (UID: \"4dc03dfd-8c9e-42a1-9677-361219162734\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t" Oct 30 00:54:23.725137 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:23.725097 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnvbk\" (UniqueName: \"kubernetes.io/projected/4dc03dfd-8c9e-42a1-9677-361219162734-kube-api-access-lnvbk\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-k659t\" (UID: \"4dc03dfd-8c9e-42a1-9677-361219162734\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t" Oct 30 00:54:23.725137 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:23.725132 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dc03dfd-8c9e-42a1-9677-361219162734-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-k659t\" (UID: \"4dc03dfd-8c9e-42a1-9677-361219162734\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t" Oct 30 00:54:23.725203 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:23.725150 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dc03dfd-8c9e-42a1-9677-361219162734-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-k659t\" (UID: \"4dc03dfd-8c9e-42a1-9677-361219162734\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t" Oct 30 00:54:23.725203 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:23.725172 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dc03dfd-8c9e-42a1-9677-361219162734-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-k659t\" (UID: \"4dc03dfd-8c9e-42a1-9677-361219162734\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t" Oct 30 00:54:23.826111 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:23.826079 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnvbk\" (UniqueName: \"kubernetes.io/projected/4dc03dfd-8c9e-42a1-9677-361219162734-kube-api-access-lnvbk\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-k659t\" (UID: \"4dc03dfd-8c9e-42a1-9677-361219162734\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t" Oct 30 00:54:23.826279 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:23.826121 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dc03dfd-8c9e-42a1-9677-361219162734-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-k659t\" (UID: \"4dc03dfd-8c9e-42a1-9677-361219162734\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t" Oct 30 00:54:23.826279 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:23.826147 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dc03dfd-8c9e-42a1-9677-361219162734-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-k659t\" (UID: \"4dc03dfd-8c9e-42a1-9677-361219162734\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t" Oct 30 00:54:23.826279 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:23.826180 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dc03dfd-8c9e-42a1-9677-361219162734-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-k659t\" (UID: \"4dc03dfd-8c9e-42a1-9677-361219162734\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t" Oct 30 00:54:23.826279 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:23.826226 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dc03dfd-8c9e-42a1-9677-361219162734-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-k659t\" (UID: \"4dc03dfd-8c9e-42a1-9677-361219162734\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t" Oct 30 00:54:23.826279 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:23.826250 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dc03dfd-8c9e-42a1-9677-361219162734-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-k659t\" (UID: \"4dc03dfd-8c9e-42a1-9677-361219162734\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t" Oct 30 00:54:23.826655 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:23.826624 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dc03dfd-8c9e-42a1-9677-361219162734-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-k659t\" (UID: \"4dc03dfd-8c9e-42a1-9677-361219162734\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t" Oct 30 00:54:23.826655 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:23.826637 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dc03dfd-8c9e-42a1-9677-361219162734-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-k659t\" (UID: \"4dc03dfd-8c9e-42a1-9677-361219162734\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t" Oct 30 00:54:23.826836 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:23.826719 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dc03dfd-8c9e-42a1-9677-361219162734-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-k659t\" (UID: \"4dc03dfd-8c9e-42a1-9677-361219162734\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t" Oct 30 00:54:23.828440 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:23.828421 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dc03dfd-8c9e-42a1-9677-361219162734-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-k659t\" (UID: \"4dc03dfd-8c9e-42a1-9677-361219162734\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t" Oct 30 00:54:23.828703 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:23.828686 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dc03dfd-8c9e-42a1-9677-361219162734-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-k659t\" (UID: \"4dc03dfd-8c9e-42a1-9677-361219162734\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t" Oct 30 00:54:23.845250 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:23.845189 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnvbk\" (UniqueName: \"kubernetes.io/projected/4dc03dfd-8c9e-42a1-9677-361219162734-kube-api-access-lnvbk\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-k659t\" (UID: \"4dc03dfd-8c9e-42a1-9677-361219162734\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t" Oct 30 00:54:23.901233 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:23.901195 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t" Oct 30 00:54:24.190971 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:24.190890 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t"] Oct 30 00:54:24.194835 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:54:24.194806 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc03dfd_8c9e_42a1_9677_361219162734.slice/crio-93b8fc50df7b740b79184190e30744eb8e588423f6360b8ddd079aa86bb1ca1d WatchSource:0}: Error finding container 93b8fc50df7b740b79184190e30744eb8e588423f6360b8ddd079aa86bb1ca1d: Status 404 returned error can't find the container with id 93b8fc50df7b740b79184190e30744eb8e588423f6360b8ddd079aa86bb1ca1d Oct 30 00:54:24.351564 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:24.351534 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t" event={"ID":"4dc03dfd-8c9e-42a1-9677-361219162734","Type":"ContainerStarted","Data":"93b8fc50df7b740b79184190e30744eb8e588423f6360b8ddd079aa86bb1ca1d"} Oct 30 00:54:25.357688 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:25.357644 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t" event={"ID":"4dc03dfd-8c9e-42a1-9677-361219162734","Type":"ContainerStarted","Data":"96c30c5114596509b7a1f3bde65e5fdbc8797687eb8696c1649f41154ec0a95e"} Oct 30 00:54:25.400754 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:25.400685 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t" podStartSLOduration=2.224606102 podStartE2EDuration="2.400665222s" totalImagesPullingTime="176.05912ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:25.399909589 +0000 UTC m=+665.715777914" watchObservedRunningTime="2026-10-30 00:54:25.400665222 +0000 UTC m=+665.716533549" Oct 30 00:54:25.896489 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:25.896457 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p"] Oct 30 00:54:25.899840 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:25.899809 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 00:54:25.901940 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:25.901919 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t" Oct 30 00:54:25.926697 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:25.926670 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 00:54:25.928125 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:25.928106 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p"] Oct 30 00:54:25.957852 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:25.957822 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:26.057755 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:26.057720 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1166fd98-968e-4fda-a0b9-0cfea1ed9e03-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p\" (UID: \"1166fd98-968e-4fda-a0b9-0cfea1ed9e03\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 00:54:26.057924 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:26.057789 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2lc2\" (UniqueName: \"kubernetes.io/projected/1166fd98-968e-4fda-a0b9-0cfea1ed9e03-kube-api-access-k2lc2\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p\" (UID: \"1166fd98-968e-4fda-a0b9-0cfea1ed9e03\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 00:54:26.057924 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:26.057891 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1166fd98-968e-4fda-a0b9-0cfea1ed9e03-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p\" (UID: \"1166fd98-968e-4fda-a0b9-0cfea1ed9e03\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 00:54:26.057924 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:26.057918 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1166fd98-968e-4fda-a0b9-0cfea1ed9e03-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p\" (UID: \"1166fd98-968e-4fda-a0b9-0cfea1ed9e03\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 00:54:26.058038 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:26.057955 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1166fd98-968e-4fda-a0b9-0cfea1ed9e03-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p\" (UID: \"1166fd98-968e-4fda-a0b9-0cfea1ed9e03\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 00:54:26.058038 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:26.058020 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1166fd98-968e-4fda-a0b9-0cfea1ed9e03-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p\" (UID: \"1166fd98-968e-4fda-a0b9-0cfea1ed9e03\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 00:54:26.158516 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:26.158430 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1166fd98-968e-4fda-a0b9-0cfea1ed9e03-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p\" (UID: \"1166fd98-968e-4fda-a0b9-0cfea1ed9e03\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 00:54:26.158516 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:26.158482 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2lc2\" (UniqueName: \"kubernetes.io/projected/1166fd98-968e-4fda-a0b9-0cfea1ed9e03-kube-api-access-k2lc2\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p\" (UID: \"1166fd98-968e-4fda-a0b9-0cfea1ed9e03\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 00:54:26.158516 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:26.158513 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1166fd98-968e-4fda-a0b9-0cfea1ed9e03-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p\" (UID: \"1166fd98-968e-4fda-a0b9-0cfea1ed9e03\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 00:54:26.158826 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:26.158528 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1166fd98-968e-4fda-a0b9-0cfea1ed9e03-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p\" (UID: \"1166fd98-968e-4fda-a0b9-0cfea1ed9e03\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 00:54:26.158826 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:26.158553 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1166fd98-968e-4fda-a0b9-0cfea1ed9e03-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p\" (UID: \"1166fd98-968e-4fda-a0b9-0cfea1ed9e03\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 00:54:26.158826 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:26.158586 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1166fd98-968e-4fda-a0b9-0cfea1ed9e03-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p\" (UID: \"1166fd98-968e-4fda-a0b9-0cfea1ed9e03\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 00:54:26.158983 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:26.158954 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1166fd98-968e-4fda-a0b9-0cfea1ed9e03-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p\" (UID: \"1166fd98-968e-4fda-a0b9-0cfea1ed9e03\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 00:54:26.159041 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:26.158982 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1166fd98-968e-4fda-a0b9-0cfea1ed9e03-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p\" (UID: \"1166fd98-968e-4fda-a0b9-0cfea1ed9e03\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 00:54:26.159077 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:26.159032 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1166fd98-968e-4fda-a0b9-0cfea1ed9e03-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p\" (UID: \"1166fd98-968e-4fda-a0b9-0cfea1ed9e03\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 00:54:26.160818 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:26.160795 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1166fd98-968e-4fda-a0b9-0cfea1ed9e03-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p\" (UID: \"1166fd98-968e-4fda-a0b9-0cfea1ed9e03\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 00:54:26.161195 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:26.161175 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1166fd98-968e-4fda-a0b9-0cfea1ed9e03-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p\" (UID: \"1166fd98-968e-4fda-a0b9-0cfea1ed9e03\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 00:54:26.178826 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:26.178802 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2lc2\" (UniqueName: \"kubernetes.io/projected/1166fd98-968e-4fda-a0b9-0cfea1ed9e03-kube-api-access-k2lc2\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p\" (UID: \"1166fd98-968e-4fda-a0b9-0cfea1ed9e03\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 00:54:26.211466 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:26.211430 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 00:54:26.503307 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:26.503273 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p"] Oct 30 00:54:26.507783 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:54:26.507743 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1166fd98_968e_4fda_a0b9_0cfea1ed9e03.slice/crio-37ee06eb075c73967ff7b8bd2d32fde29b1901244ce5686b10aafa41f3a61df4 WatchSource:0}: Error finding container 37ee06eb075c73967ff7b8bd2d32fde29b1901244ce5686b10aafa41f3a61df4: Status 404 returned error can't find the container with id 37ee06eb075c73967ff7b8bd2d32fde29b1901244ce5686b10aafa41f3a61df4 Oct 30 00:54:27.284686 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:27.284640 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh" Oct 30 00:54:27.287448 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:27.287418 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:27.297573 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:27.297548 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh" Oct 30 00:54:27.368914 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:27.368876 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" event={"ID":"1166fd98-968e-4fda-a0b9-0cfea1ed9e03","Type":"ContainerStarted","Data":"a8749e58322e77c66f6554eebc14a50a7c567939bfdd0f648461ae494a61eeaf"} Oct 30 00:54:27.369097 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:27.368926 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" event={"ID":"1166fd98-968e-4fda-a0b9-0cfea1ed9e03","Type":"ContainerStarted","Data":"37ee06eb075c73967ff7b8bd2d32fde29b1901244ce5686b10aafa41f3a61df4"} Oct 30 00:54:27.385317 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:27.385280 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-696nh" Oct 30 00:54:27.410059 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:27.410003 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" podStartSLOduration=2.2191850889999998 podStartE2EDuration="2.409989343s" totalImagesPullingTime="190.804254ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:27.40899311 +0000 UTC m=+667.724861436" watchObservedRunningTime="2026-10-30 00:54:27.409989343 +0000 UTC m=+667.725857646" Oct 30 00:54:28.212536 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:28.212505 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 00:54:28.704575 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:28.704537 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd" Oct 30 00:54:28.717289 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:28.717259 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd" Oct 30 00:54:29.388471 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:29.388442 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kzbhd" Oct 30 00:54:29.394242 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:29.394216 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:33.902239 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:33.902202 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t" Oct 30 00:54:33.914582 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:33.914554 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t" Oct 30 00:54:34.411393 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:34.411363 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-k659t" Oct 30 00:54:36.212356 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:36.212320 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 00:54:36.224824 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:36.224798 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 00:54:36.420328 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:36.420295 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 00:54:43.821291 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:43.821255 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dfd774998-7khzv"] Oct 30 00:54:43.824918 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:43.824895 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dfd774998-7khzv" Oct 30 00:54:43.841841 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:43.841817 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dfd774998-7khzv"] Oct 30 00:54:43.937578 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:43.937548 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqpsp\" (UniqueName: \"kubernetes.io/projected/ef5c2a40-f573-4613-b77f-6b8b72dfc1cc-kube-api-access-tqpsp\") pod \"authorino-5dfd774998-7khzv\" (UID: \"ef5c2a40-f573-4613-b77f-6b8b72dfc1cc\") " pod="kuadrant-system/authorino-5dfd774998-7khzv" Oct 30 00:54:43.937578 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:43.937581 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef5c2a40-f573-4613-b77f-6b8b72dfc1cc-tls-cert\") pod \"authorino-5dfd774998-7khzv\" (UID: \"ef5c2a40-f573-4613-b77f-6b8b72dfc1cc\") " pod="kuadrant-system/authorino-5dfd774998-7khzv" Oct 30 00:54:44.038596 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:44.038559 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqpsp\" (UniqueName: \"kubernetes.io/projected/ef5c2a40-f573-4613-b77f-6b8b72dfc1cc-kube-api-access-tqpsp\") pod \"authorino-5dfd774998-7khzv\" (UID: \"ef5c2a40-f573-4613-b77f-6b8b72dfc1cc\") " pod="kuadrant-system/authorino-5dfd774998-7khzv" Oct 30 00:54:44.038596 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:44.038602 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef5c2a40-f573-4613-b77f-6b8b72dfc1cc-tls-cert\") pod \"authorino-5dfd774998-7khzv\" (UID: \"ef5c2a40-f573-4613-b77f-6b8b72dfc1cc\") " pod="kuadrant-system/authorino-5dfd774998-7khzv" Oct 30 00:54:44.041050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:44.041027 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef5c2a40-f573-4613-b77f-6b8b72dfc1cc-tls-cert\") pod \"authorino-5dfd774998-7khzv\" (UID: \"ef5c2a40-f573-4613-b77f-6b8b72dfc1cc\") " pod="kuadrant-system/authorino-5dfd774998-7khzv" Oct 30 00:54:44.058056 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:44.058028 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqpsp\" (UniqueName: \"kubernetes.io/projected/ef5c2a40-f573-4613-b77f-6b8b72dfc1cc-kube-api-access-tqpsp\") pod \"authorino-5dfd774998-7khzv\" (UID: \"ef5c2a40-f573-4613-b77f-6b8b72dfc1cc\") " pod="kuadrant-system/authorino-5dfd774998-7khzv" Oct 30 00:54:44.136651 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:44.136566 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dfd774998-7khzv" Oct 30 00:54:44.252625 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:44.252588 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:44.426631 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:44.426550 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dfd774998-7khzv"] Oct 30 00:54:44.430609 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:54:44.430565 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef5c2a40_f573_4613_b77f_6b8b72dfc1cc.slice/crio-1284d6ec163f788ee94a0ac21aa6bdafb5d54d6b246731204f893dc038d2fac6 WatchSource:0}: Error finding container 1284d6ec163f788ee94a0ac21aa6bdafb5d54d6b246731204f893dc038d2fac6: Status 404 returned error can't find the container with id 1284d6ec163f788ee94a0ac21aa6bdafb5d54d6b246731204f893dc038d2fac6 Oct 30 00:54:44.442439 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:44.442409 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dfd774998-7khzv" event={"ID":"ef5c2a40-f573-4613-b77f-6b8b72dfc1cc","Type":"ContainerStarted","Data":"1284d6ec163f788ee94a0ac21aa6bdafb5d54d6b246731204f893dc038d2fac6"} Oct 30 00:54:44.545992 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:44.545947 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:45.450579 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:45.450538 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dfd774998-7khzv" event={"ID":"ef5c2a40-f573-4613-b77f-6b8b72dfc1cc","Type":"ContainerStarted","Data":"be5582fca5e081fe0ed94a83107aaacd3a90cf849e1f4fcd3aabb7aa982e4fa2"} Oct 30 00:54:45.489495 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:45.489440 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5dfd774998-7khzv" podStartSLOduration=2.090925581 podStartE2EDuration="2.489421229s" totalImagesPullingTime="398.495648ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:45.488148314 +0000 UTC m=+685.804016638" watchObservedRunningTime="2026-10-30 00:54:45.489421229 +0000 UTC m=+685.805289555" Oct 30 00:54:45.515147 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:45.515100 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dfd97fb6f-4rk72"] Oct 30 00:54:45.515433 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:45.515379 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5dfd97fb6f-4rk72" podUID="839c0725-66a2-4100-bd25-da52516a001c" containerName="authorino" containerID="cri-o://106ce3e5a96bfbe36b3b532cf08d7a7d45368ccf8ae30c47986f07d6c4bc75be" gracePeriod=30 Oct 30 00:54:45.780899 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:45.780868 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dfd97fb6f-4rk72" Oct 30 00:54:45.857366 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:45.857328 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/839c0725-66a2-4100-bd25-da52516a001c-kube-api-access-wzp8f\" (UniqueName: \"kubernetes.io/projected/839c0725-66a2-4100-bd25-da52516a001c-kube-api-access-wzp8f\") pod \"839c0725-66a2-4100-bd25-da52516a001c\" (UID: \"839c0725-66a2-4100-bd25-da52516a001c\") " Oct 30 00:54:45.857511 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:45.857475 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/839c0725-66a2-4100-bd25-da52516a001c-tls-cert\" (UniqueName: \"kubernetes.io/secret/839c0725-66a2-4100-bd25-da52516a001c-tls-cert\") pod \"839c0725-66a2-4100-bd25-da52516a001c\" (UID: \"839c0725-66a2-4100-bd25-da52516a001c\") " Oct 30 00:54:45.860273 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:45.860236 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839c0725-66a2-4100-bd25-da52516a001c-kube-api-access-wzp8f" pod "839c0725-66a2-4100-bd25-da52516a001c" (UID: "839c0725-66a2-4100-bd25-da52516a001c"). InnerVolumeSpecName "kube-api-access-wzp8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 00:54:45.867939 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:45.867906 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/839c0725-66a2-4100-bd25-da52516a001c-tls-cert" pod "839c0725-66a2-4100-bd25-da52516a001c" (UID: "839c0725-66a2-4100-bd25-da52516a001c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 00:54:45.958207 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:45.958169 5300 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/839c0725-66a2-4100-bd25-da52516a001c-tls-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:54:45.958207 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:45.958202 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzp8f\" (UniqueName: \"kubernetes.io/projected/839c0725-66a2-4100-bd25-da52516a001c-kube-api-access-wzp8f\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 00:54:46.456271 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:46.456185 5300 generic.go:356] "Generic (PLEG): container finished" podID="839c0725-66a2-4100-bd25-da52516a001c" containerID="106ce3e5a96bfbe36b3b532cf08d7a7d45368ccf8ae30c47986f07d6c4bc75be" exitCode=0 Oct 30 00:54:46.456271 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:46.456228 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dfd97fb6f-4rk72" Oct 30 00:54:46.456702 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:46.456268 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dfd97fb6f-4rk72" event={"ID":"839c0725-66a2-4100-bd25-da52516a001c","Type":"ContainerDied","Data":"106ce3e5a96bfbe36b3b532cf08d7a7d45368ccf8ae30c47986f07d6c4bc75be"} Oct 30 00:54:46.456702 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:46.456314 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dfd97fb6f-4rk72" event={"ID":"839c0725-66a2-4100-bd25-da52516a001c","Type":"ContainerDied","Data":"fc891d25ba5b79cc9253adb5180f6a0b2f74a7ed0cdefc2302a9ca2b756a30c3"} Oct 30 00:54:46.456702 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:46.456336 5300 scope.go:122] "RemoveContainer" containerID="106ce3e5a96bfbe36b3b532cf08d7a7d45368ccf8ae30c47986f07d6c4bc75be" Oct 30 00:54:46.468179 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:46.468157 5300 scope.go:122] "RemoveContainer" containerID="106ce3e5a96bfbe36b3b532cf08d7a7d45368ccf8ae30c47986f07d6c4bc75be" Oct 30 00:54:46.468568 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:54:46.468540 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"106ce3e5a96bfbe36b3b532cf08d7a7d45368ccf8ae30c47986f07d6c4bc75be\": container with ID starting with 106ce3e5a96bfbe36b3b532cf08d7a7d45368ccf8ae30c47986f07d6c4bc75be not found: ID does not exist" containerID="106ce3e5a96bfbe36b3b532cf08d7a7d45368ccf8ae30c47986f07d6c4bc75be" Oct 30 00:54:46.468645 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:46.468577 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"106ce3e5a96bfbe36b3b532cf08d7a7d45368ccf8ae30c47986f07d6c4bc75be"} err="failed to get container status \"106ce3e5a96bfbe36b3b532cf08d7a7d45368ccf8ae30c47986f07d6c4bc75be\": rpc error: code = NotFound desc = could not find container \"106ce3e5a96bfbe36b3b532cf08d7a7d45368ccf8ae30c47986f07d6c4bc75be\": container with ID starting with 106ce3e5a96bfbe36b3b532cf08d7a7d45368ccf8ae30c47986f07d6c4bc75be not found: ID does not exist" Oct 30 00:54:46.506348 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:46.506310 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dfd97fb6f-4rk72"] Oct 30 00:54:46.518810 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:46.518756 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dfd97fb6f-4rk72"] Oct 30 00:54:47.819494 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:47.819462 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="839c0725-66a2-4100-bd25-da52516a001c" path="/var/lib/kubelet/pods/839c0725-66a2-4100-bd25-da52516a001c/volumes" Oct 30 00:54:48.923780 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:48.923351 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:49.008287 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:49.008253 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:54:49.194350 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:49.194119 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-nkqms"] Oct 30 00:54:49.194568 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:49.194552 5300 cpu_manager.go:414] "containerMap: removing container" podUID="839c0725-66a2-4100-bd25-da52516a001c" containerName="authorino" Oct 30 00:54:49.194615 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:49.194569 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="839c0725-66a2-4100-bd25-da52516a001c" containerName="authorino" Oct 30 00:54:49.194661 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:49.194651 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="839c0725-66a2-4100-bd25-da52516a001c" containerName="authorino" Oct 30 00:54:49.200775 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:49.200722 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-nkqms" Oct 30 00:54:49.218455 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:49.218430 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-9fnng\"" Oct 30 00:54:49.232492 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:49.232463 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-nkqms"] Oct 30 00:54:49.289140 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:49.289103 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmbqf\" (UniqueName: \"kubernetes.io/projected/7817f490-6fc5-4d17-8acc-f83273bd9a5c-kube-api-access-rmbqf\") pod \"payload-processing-5f78687d5b-nkqms\" (UID: \"7817f490-6fc5-4d17-8acc-f83273bd9a5c\") " pod="openshift-ingress/payload-processing-5f78687d5b-nkqms" Oct 30 00:54:49.289331 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:49.289249 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7817f490-6fc5-4d17-8acc-f83273bd9a5c-plugins-config-volume\") pod \"payload-processing-5f78687d5b-nkqms\" (UID: \"7817f490-6fc5-4d17-8acc-f83273bd9a5c\") " pod="openshift-ingress/payload-processing-5f78687d5b-nkqms" Oct 30 00:54:49.308458 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:49.308422 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 00:54:49.390436 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:49.390399 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7817f490-6fc5-4d17-8acc-f83273bd9a5c-plugins-config-volume\") pod \"payload-processing-5f78687d5b-nkqms\" (UID: \"7817f490-6fc5-4d17-8acc-f83273bd9a5c\") " pod="openshift-ingress/payload-processing-5f78687d5b-nkqms" Oct 30 00:54:49.390621 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:49.390455 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmbqf\" (UniqueName: \"kubernetes.io/projected/7817f490-6fc5-4d17-8acc-f83273bd9a5c-kube-api-access-rmbqf\") pod \"payload-processing-5f78687d5b-nkqms\" (UID: \"7817f490-6fc5-4d17-8acc-f83273bd9a5c\") " pod="openshift-ingress/payload-processing-5f78687d5b-nkqms" Oct 30 00:54:49.391104 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:49.391077 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7817f490-6fc5-4d17-8acc-f83273bd9a5c-plugins-config-volume\") pod \"payload-processing-5f78687d5b-nkqms\" (UID: \"7817f490-6fc5-4d17-8acc-f83273bd9a5c\") " pod="openshift-ingress/payload-processing-5f78687d5b-nkqms" Oct 30 00:54:49.410634 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:49.410600 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmbqf\" (UniqueName: \"kubernetes.io/projected/7817f490-6fc5-4d17-8acc-f83273bd9a5c-kube-api-access-rmbqf\") pod \"payload-processing-5f78687d5b-nkqms\" (UID: \"7817f490-6fc5-4d17-8acc-f83273bd9a5c\") " pod="openshift-ingress/payload-processing-5f78687d5b-nkqms" Oct 30 00:54:49.514603 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:49.514565 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-nkqms" Oct 30 00:54:49.807988 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:49.807915 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-nkqms"] Oct 30 00:54:49.811531 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:54:49.811498 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7817f490_6fc5_4d17_8acc_f83273bd9a5c.slice/crio-babc173750fe5faa7170291bfc6e0bacb685e018b3ffe2953895a6646c9ffdcf WatchSource:0}: Error finding container babc173750fe5faa7170291bfc6e0bacb685e018b3ffe2953895a6646c9ffdcf: Status 404 returned error can't find the container with id babc173750fe5faa7170291bfc6e0bacb685e018b3ffe2953895a6646c9ffdcf Oct 30 00:54:50.495613 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:50.495574 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-nkqms" event={"ID":"7817f490-6fc5-4d17-8acc-f83273bd9a5c","Type":"ContainerStarted","Data":"babc173750fe5faa7170291bfc6e0bacb685e018b3ffe2953895a6646c9ffdcf"} Oct 30 00:54:51.596990 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:54:51.596941 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:54:52.306573 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.306538 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8"] Oct 30 00:54:52.310447 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.310420 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.324803 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.324780 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 00:54:52.324936 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.324832 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v5fml\"" Oct 30 00:54:52.338592 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.338566 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8"] Oct 30 00:54:52.416837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.416804 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-9x6g8\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.417036 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.416845 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-9x6g8\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.417036 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.416866 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-9x6g8\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.417036 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.416881 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfwfz\" (UniqueName: \"kubernetes.io/projected/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-kube-api-access-bfwfz\") pod \"maas-default-gateway-openshift-default-67bf986f99-9x6g8\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.417191 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.417024 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9x6g8\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.417191 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.417057 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-9x6g8\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.417191 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.417077 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9x6g8\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.417338 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.417184 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-9x6g8\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.417338 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.417250 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-9x6g8\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.506925 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.506890 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-nkqms" event={"ID":"7817f490-6fc5-4d17-8acc-f83273bd9a5c","Type":"ContainerStarted","Data":"99e21deb02b3093426d8792ea43e5984b48e6eb5ae1dd362c747da690e940a20"} Oct 30 00:54:52.507122 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.506953 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-nkqms" Oct 30 00:54:52.518442 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.518418 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-9x6g8\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.518541 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.518456 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-9x6g8\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.518541 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.518490 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-9x6g8\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.518541 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.518523 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-9x6g8\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.518692 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.518643 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-9x6g8\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.518692 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.518679 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfwfz\" (UniqueName: \"kubernetes.io/projected/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-kube-api-access-bfwfz\") pod \"maas-default-gateway-openshift-default-67bf986f99-9x6g8\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.518825 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.518741 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9x6g8\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.518825 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.518784 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-9x6g8\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.518825 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.518811 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9x6g8\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.519228 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.519182 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-9x6g8\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.519351 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.519276 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9x6g8\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.519351 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.519238 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-9x6g8\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.519351 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.519219 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-9x6g8\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.519351 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.519342 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-9x6g8\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.521023 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.520995 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-9x6g8\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.521175 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.521156 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-9x6g8\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.537218 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.537194 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-9x6g8\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.537503 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.537481 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfwfz\" (UniqueName: \"kubernetes.io/projected/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-kube-api-access-bfwfz\") pod \"maas-default-gateway-openshift-default-67bf986f99-9x6g8\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.544623 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.544572 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-nkqms" podStartSLOduration=1.653988724 podStartE2EDuration="3.544556475s" totalImagesPullingTime="1.890567751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:52.543328759 +0000 UTC m=+692.859197085" watchObservedRunningTime="2026-10-30 00:54:52.544556475 +0000 UTC m=+692.860424800" Oct 30 00:54:52.623519 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.623425 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:52.943500 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:52.943411 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8"] Oct 30 00:54:52.947480 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:54:52.947449 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3f3856_e654_4c1e_b1d7_35903a4dfe89.slice/crio-58f74ff226bdb1015636ecc26e1f9fac9b3408a95d2d447dc809bf0a92597b60 WatchSource:0}: Error finding container 58f74ff226bdb1015636ecc26e1f9fac9b3408a95d2d447dc809bf0a92597b60: Status 404 returned error can't find the container with id 58f74ff226bdb1015636ecc26e1f9fac9b3408a95d2d447dc809bf0a92597b60 Oct 30 00:54:53.512988 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:53.512950 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" event={"ID":"3c3f3856-e654-4c1e-b1d7-35903a4dfe89","Type":"ContainerStarted","Data":"58f74ff226bdb1015636ecc26e1f9fac9b3408a95d2d447dc809bf0a92597b60"} Oct 30 00:54:56.210160 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:56.210122 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:54:56.210446 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:56.210190 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:54:56.210446 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:56.210221 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:54:56.530009 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:56.529964 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" event={"ID":"3c3f3856-e654-4c1e-b1d7-35903a4dfe89","Type":"ContainerStarted","Data":"8ecc1b38721a00dd3a706be0c63b43f4cd0a34963995950d3e03dcf8cee702f7"} Oct 30 00:54:56.596183 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:56.596116 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" podStartSLOduration=1.336280146 podStartE2EDuration="4.596097086s" totalImagesPullingTime="3.25981694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:56.582216715 +0000 UTC m=+696.898085067" watchObservedRunningTime="2026-10-30 00:54:56.596097086 +0000 UTC m=+696.911965412" Oct 30 00:54:56.624055 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:56.624024 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:54:57.639039 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:57.638998 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9x6g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 00:54:57.639663 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:57.639615 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" podUID="3c3f3856-e654-4c1e-b1d7-35903a4dfe89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 00:54:58.639785 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:58.639173 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9x6g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 00:54:58.639785 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:58.639237 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" podUID="3c3f3856-e654-4c1e-b1d7-35903a4dfe89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 00:54:59.638787 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:59.637219 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9x6g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 00:54:59.638787 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:54:59.637303 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" podUID="3c3f3856-e654-4c1e-b1d7-35903a4dfe89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 00:55:00.634785 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:00.630182 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9x6g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 00:55:00.634785 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:00.630248 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" podUID="3c3f3856-e654-4c1e-b1d7-35903a4dfe89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 00:55:01.641804 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:01.637136 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9x6g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 00:55:01.641804 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:01.637202 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" podUID="3c3f3856-e654-4c1e-b1d7-35903a4dfe89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 00:55:02.623987 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:02.623954 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9x6g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 00:55:02.624168 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:02.624006 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" podUID="3c3f3856-e654-4c1e-b1d7-35903a4dfe89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 00:55:03.516434 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:03.516402 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-nkqms" Oct 30 00:55:03.624032 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:03.624001 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-9x6g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 00:55:03.624247 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:03.624055 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" podUID="3c3f3856-e654-4c1e-b1d7-35903a4dfe89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 00:55:04.628208 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:04.628173 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:55:04.628612 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:04.628566 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:55:04.629308 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:04.629288 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 00:55:07.359460 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.359424 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6"] Oct 30 00:55:07.362918 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.362890 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.386111 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.386078 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm"] Oct 30 00:55:07.389416 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.389393 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.391946 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.391922 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6"] Oct 30 00:55:07.411794 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.411731 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm"] Oct 30 00:55:07.462631 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.462601 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-4zkk6\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.462631 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.462633 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f157e0-a09c-46c7-bd51-43bd5567b9ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwmfm\" (UID: \"66f157e0-a09c-46c7-bd51-43bd5567b9ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.462861 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.462656 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e890163-36d6-42fb-bae0-3140dbe32425-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-4zkk6\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.462861 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.462733 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f157e0-a09c-46c7-bd51-43bd5567b9ab-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwmfm\" (UID: \"66f157e0-a09c-46c7-bd51-43bd5567b9ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.462861 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.462795 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e890163-36d6-42fb-bae0-3140dbe32425-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-4zkk6\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.462976 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.462890 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-4zkk6\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.462976 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.462934 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66f157e0-a09c-46c7-bd51-43bd5567b9ab-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwmfm\" (UID: \"66f157e0-a09c-46c7-bd51-43bd5567b9ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.462976 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.462953 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f157e0-a09c-46c7-bd51-43bd5567b9ab-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwmfm\" (UID: \"66f157e0-a09c-46c7-bd51-43bd5567b9ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.463077 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.462981 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27vht\" (UniqueName: \"kubernetes.io/projected/0e890163-36d6-42fb-bae0-3140dbe32425-kube-api-access-27vht\") pod \"maas-default-gateway-openshift-default-67bf986f99-4zkk6\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.463077 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.463019 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-4zkk6\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.463077 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.463050 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-4zkk6\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.463179 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.463097 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f157e0-a09c-46c7-bd51-43bd5567b9ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwmfm\" (UID: \"66f157e0-a09c-46c7-bd51-43bd5567b9ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.463179 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.463116 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f157e0-a09c-46c7-bd51-43bd5567b9ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwmfm\" (UID: \"66f157e0-a09c-46c7-bd51-43bd5567b9ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.463179 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.463132 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e890163-36d6-42fb-bae0-3140dbe32425-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-4zkk6\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.463179 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.463151 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f157e0-a09c-46c7-bd51-43bd5567b9ab-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwmfm\" (UID: \"66f157e0-a09c-46c7-bd51-43bd5567b9ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.463316 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.463184 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-4zkk6\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.463316 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.463248 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhn7x\" (UniqueName: \"kubernetes.io/projected/66f157e0-a09c-46c7-bd51-43bd5567b9ab-kube-api-access-fhn7x\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwmfm\" (UID: \"66f157e0-a09c-46c7-bd51-43bd5567b9ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.463316 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.463267 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f157e0-a09c-46c7-bd51-43bd5567b9ab-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwmfm\" (UID: \"66f157e0-a09c-46c7-bd51-43bd5567b9ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.564748 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.564712 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-4zkk6\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.564942 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.564755 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66f157e0-a09c-46c7-bd51-43bd5567b9ab-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwmfm\" (UID: \"66f157e0-a09c-46c7-bd51-43bd5567b9ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.564942 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.564809 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f157e0-a09c-46c7-bd51-43bd5567b9ab-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwmfm\" (UID: \"66f157e0-a09c-46c7-bd51-43bd5567b9ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.564942 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.564840 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27vht\" (UniqueName: \"kubernetes.io/projected/0e890163-36d6-42fb-bae0-3140dbe32425-kube-api-access-27vht\") pod \"maas-default-gateway-openshift-default-67bf986f99-4zkk6\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.564942 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.564864 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-4zkk6\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.564942 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.564910 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-4zkk6\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.565258 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.564952 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f157e0-a09c-46c7-bd51-43bd5567b9ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwmfm\" (UID: \"66f157e0-a09c-46c7-bd51-43bd5567b9ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.565258 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.564981 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f157e0-a09c-46c7-bd51-43bd5567b9ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwmfm\" (UID: \"66f157e0-a09c-46c7-bd51-43bd5567b9ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.565258 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.565010 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e890163-36d6-42fb-bae0-3140dbe32425-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-4zkk6\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.565258 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.565042 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f157e0-a09c-46c7-bd51-43bd5567b9ab-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwmfm\" (UID: \"66f157e0-a09c-46c7-bd51-43bd5567b9ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.565258 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.565071 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-4zkk6\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.565258 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.565124 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhn7x\" (UniqueName: \"kubernetes.io/projected/66f157e0-a09c-46c7-bd51-43bd5567b9ab-kube-api-access-fhn7x\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwmfm\" (UID: \"66f157e0-a09c-46c7-bd51-43bd5567b9ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.565258 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.565157 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f157e0-a09c-46c7-bd51-43bd5567b9ab-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwmfm\" (UID: \"66f157e0-a09c-46c7-bd51-43bd5567b9ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.565258 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.565197 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-4zkk6\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.565258 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.565210 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-4zkk6\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.565258 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.565213 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f157e0-a09c-46c7-bd51-43bd5567b9ab-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwmfm\" (UID: \"66f157e0-a09c-46c7-bd51-43bd5567b9ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.565258 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.565241 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f157e0-a09c-46c7-bd51-43bd5567b9ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwmfm\" (UID: \"66f157e0-a09c-46c7-bd51-43bd5567b9ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.565845 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.565277 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e890163-36d6-42fb-bae0-3140dbe32425-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-4zkk6\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.565845 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.565318 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f157e0-a09c-46c7-bd51-43bd5567b9ab-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwmfm\" (UID: \"66f157e0-a09c-46c7-bd51-43bd5567b9ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.565845 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.565350 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e890163-36d6-42fb-bae0-3140dbe32425-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-4zkk6\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.565845 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.565493 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f157e0-a09c-46c7-bd51-43bd5567b9ab-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwmfm\" (UID: \"66f157e0-a09c-46c7-bd51-43bd5567b9ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.565845 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.565571 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-4zkk6\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.565845 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.565654 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f157e0-a09c-46c7-bd51-43bd5567b9ab-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwmfm\" (UID: \"66f157e0-a09c-46c7-bd51-43bd5567b9ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.566183 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.566110 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f157e0-a09c-46c7-bd51-43bd5567b9ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwmfm\" (UID: \"66f157e0-a09c-46c7-bd51-43bd5567b9ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.566183 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.566158 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f157e0-a09c-46c7-bd51-43bd5567b9ab-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwmfm\" (UID: \"66f157e0-a09c-46c7-bd51-43bd5567b9ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.566356 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.566334 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-4zkk6\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.566446 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.566430 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-4zkk6\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.566886 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.566861 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e890163-36d6-42fb-bae0-3140dbe32425-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-4zkk6\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.567634 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.567611 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e890163-36d6-42fb-bae0-3140dbe32425-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-4zkk6\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.567721 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.567643 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f157e0-a09c-46c7-bd51-43bd5567b9ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwmfm\" (UID: \"66f157e0-a09c-46c7-bd51-43bd5567b9ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.567784 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.567723 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-4zkk6\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.567896 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.567879 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f157e0-a09c-46c7-bd51-43bd5567b9ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwmfm\" (UID: \"66f157e0-a09c-46c7-bd51-43bd5567b9ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.584026 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.584000 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66f157e0-a09c-46c7-bd51-43bd5567b9ab-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwmfm\" (UID: \"66f157e0-a09c-46c7-bd51-43bd5567b9ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.584154 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.584138 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27vht\" (UniqueName: \"kubernetes.io/projected/0e890163-36d6-42fb-bae0-3140dbe32425-kube-api-access-27vht\") pod \"maas-default-gateway-openshift-default-67bf986f99-4zkk6\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.584483 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.584468 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e890163-36d6-42fb-bae0-3140dbe32425-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-4zkk6\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.585031 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.585010 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhn7x\" (UniqueName: \"kubernetes.io/projected/66f157e0-a09c-46c7-bd51-43bd5567b9ab-kube-api-access-fhn7x\") pod \"maas-default-gateway-openshift-default-67bf986f99-pwmfm\" (UID: \"66f157e0-a09c-46c7-bd51-43bd5567b9ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.675599 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.675516 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:07.703952 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.703922 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:07.984639 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.984603 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6"] Oct 30 00:55:07.988510 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:55:07.988482 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e890163_36d6_42fb_bae0_3140dbe32425.slice/crio-d2107d2683d170b396cc91eedef8baf757c7ad5bafd868179df72888504f709c WatchSource:0}: Error finding container d2107d2683d170b396cc91eedef8baf757c7ad5bafd868179df72888504f709c: Status 404 returned error can't find the container with id d2107d2683d170b396cc91eedef8baf757c7ad5bafd868179df72888504f709c Oct 30 00:55:07.991694 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.991662 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:55:07.991788 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.991733 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:55:07.991788 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:07.991776 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:55:08.022880 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:08.022857 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm"] Oct 30 00:55:08.025815 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:55:08.025793 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f157e0_a09c_46c7_bd51_43bd5567b9ab.slice/crio-b5866efe894a4abf69f6cf534832f7fa44e33827e42d4ad4bbe0e1bf6c48078c WatchSource:0}: Error finding container b5866efe894a4abf69f6cf534832f7fa44e33827e42d4ad4bbe0e1bf6c48078c: Status 404 returned error can't find the container with id b5866efe894a4abf69f6cf534832f7fa44e33827e42d4ad4bbe0e1bf6c48078c Oct 30 00:55:08.029012 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:08.028979 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:55:08.029101 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:08.029043 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:55:08.029101 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:08.029071 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:55:08.595542 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:08.595508 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" event={"ID":"0e890163-36d6-42fb-bae0-3140dbe32425","Type":"ContainerStarted","Data":"f854a3a5665ca3d70242643908783045e1a42e123a9e4512f360c7b4b6dc5f3d"} Oct 30 00:55:08.595948 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:08.595569 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" event={"ID":"0e890163-36d6-42fb-bae0-3140dbe32425","Type":"ContainerStarted","Data":"d2107d2683d170b396cc91eedef8baf757c7ad5bafd868179df72888504f709c"} Oct 30 00:55:08.597125 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:08.597103 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" event={"ID":"66f157e0-a09c-46c7-bd51-43bd5567b9ab","Type":"ContainerStarted","Data":"263a025e561659abf8275b95cdc73674ab9da19f514355c6c432c7d69a8c21c5"} Oct 30 00:55:08.597176 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:08.597149 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" event={"ID":"66f157e0-a09c-46c7-bd51-43bd5567b9ab","Type":"ContainerStarted","Data":"b5866efe894a4abf69f6cf534832f7fa44e33827e42d4ad4bbe0e1bf6c48078c"} Oct 30 00:55:08.676028 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:08.675982 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:08.704461 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:08.704425 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:08.709158 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:08.709115 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" podStartSLOduration=1.709103107 podStartE2EDuration="1.709103107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:08.708488815 +0000 UTC m=+709.024357141" watchObservedRunningTime="2026-10-30 00:55:08.709103107 +0000 UTC m=+709.024971433" Oct 30 00:55:08.710108 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:08.710027 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" podStartSLOduration=1.709998289 podStartE2EDuration="1.709998289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:08.665602938 +0000 UTC m=+708.981471264" watchObservedRunningTime="2026-10-30 00:55:08.709998289 +0000 UTC m=+709.025866611" Oct 30 00:55:09.678039 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:09.678003 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4zkk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 00:55:09.678439 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:09.678055 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" podUID="0e890163-36d6-42fb-bae0-3140dbe32425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 00:55:09.706162 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:09.706122 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pwmfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 00:55:09.706312 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:09.706177 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" podUID="66f157e0-a09c-46c7-bd51-43bd5567b9ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 00:55:10.676289 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:10.676247 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4zkk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 00:55:10.676483 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:10.676300 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" podUID="0e890163-36d6-42fb-bae0-3140dbe32425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 00:55:10.704203 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:10.704175 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pwmfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 00:55:10.704567 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:10.704225 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" podUID="66f157e0-a09c-46c7-bd51-43bd5567b9ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 00:55:11.437601 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:11.437426 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:11.676286 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:11.676254 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4zkk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 00:55:11.676520 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:11.676499 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" podUID="0e890163-36d6-42fb-bae0-3140dbe32425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 00:55:11.704511 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:11.704435 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pwmfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 00:55:11.704970 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:11.704946 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" podUID="66f157e0-a09c-46c7-bd51-43bd5567b9ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 00:55:12.676322 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:12.676287 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4zkk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 00:55:12.676497 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:12.676340 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" podUID="0e890163-36d6-42fb-bae0-3140dbe32425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 00:55:12.705231 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:12.705195 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pwmfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 00:55:12.705609 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:12.705245 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" podUID="66f157e0-a09c-46c7-bd51-43bd5567b9ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 00:55:13.676866 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:13.676827 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4zkk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 00:55:13.677063 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:13.676886 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" podUID="0e890163-36d6-42fb-bae0-3140dbe32425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 00:55:13.704817 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:13.704753 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pwmfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 00:55:13.704978 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:13.704830 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" podUID="66f157e0-a09c-46c7-bd51-43bd5567b9ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 00:55:13.747235 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:13.747202 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:14.228339 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:14.228298 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:14.676987 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:14.676887 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4zkk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 00:55:14.676987 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:14.676942 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" podUID="0e890163-36d6-42fb-bae0-3140dbe32425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 00:55:14.704451 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:14.704423 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pwmfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 00:55:14.704622 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:14.704471 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" podUID="66f157e0-a09c-46c7-bd51-43bd5567b9ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 00:55:15.765439 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:15.765393 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:15.865933 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:15.865888 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:16.461406 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:16.461369 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:16.461795 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:16.461749 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:16.463021 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:16.462991 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pwmfm" Oct 30 00:55:16.467317 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:16.467289 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:16.467796 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:16.467777 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:16.468295 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:16.468277 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 00:55:17.253324 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:17.253289 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:17.364657 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:17.364620 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:21.592177 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:55:21.592141 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:55:28.364868 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:28.364826 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:28.466096 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:28.466061 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:29.517309 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:29.517268 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:49.620678 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:49.620642 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:49.716135 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:49.716105 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:50.103891 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.103837 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:50.237680 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.237649 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv"] Oct 30 00:55:50.241628 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.241605 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.257918 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.257887 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-c104b8-openshift-default-dockercfg-m9r8p\"" Oct 30 00:55:50.272218 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.272179 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv"] Oct 30 00:55:50.292084 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.292051 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:50.320941 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.320912 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78eeb03d-9184-4a3d-af85-b8391731e828-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.321087 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.320948 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.321087 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.321032 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.321087 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.321065 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc5tb\" (UniqueName: \"kubernetes.io/projected/78eeb03d-9184-4a3d-af85-b8391731e828-kube-api-access-tc5tb\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.321207 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.321164 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.321207 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.321187 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.321281 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.321228 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78eeb03d-9184-4a3d-af85-b8391731e828-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.321281 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.321268 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78eeb03d-9184-4a3d-af85-b8391731e828-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.321356 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.321316 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.378750 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.378670 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt"] Oct 30 00:55:50.382265 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.382242 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.392023 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.391998 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:50.405922 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.405896 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt"] Oct 30 00:55:50.421903 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.421877 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.422060 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.421917 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.422060 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.421972 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78eeb03d-9184-4a3d-af85-b8391731e828-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.422060 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.422007 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78eeb03d-9184-4a3d-af85-b8391731e828-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.422060 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.422034 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.422269 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.422068 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78eeb03d-9184-4a3d-af85-b8391731e828-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.422376 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.422350 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.422462 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.422352 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.422462 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.422446 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.422569 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.422494 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc5tb\" (UniqueName: \"kubernetes.io/projected/78eeb03d-9184-4a3d-af85-b8391731e828-kube-api-access-tc5tb\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.422655 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.422624 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.422796 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.422755 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78eeb03d-9184-4a3d-af85-b8391731e828-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.422879 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.422792 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.422994 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.422977 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.424796 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.424221 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.424796 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.424593 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78eeb03d-9184-4a3d-af85-b8391731e828-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.435836 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.435817 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-82b647-openshift-default-dockercfg-w4zsm\"" Oct 30 00:55:50.441091 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.441071 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78eeb03d-9184-4a3d-af85-b8391731e828-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.441234 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.441218 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc5tb\" (UniqueName: \"kubernetes.io/projected/78eeb03d-9184-4a3d-af85-b8391731e828-kube-api-access-tc5tb\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.494065 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.494037 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:50.523450 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.523404 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-workload-certs\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.523450 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.523450 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f99a456-9f20-4d79-950e-a79b016f5043-istio-token\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.523656 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.523491 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f99a456-9f20-4d79-950e-a79b016f5043-istio-podinfo\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.523656 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.523641 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-workload-socket\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.523803 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.523694 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-istio-envoy\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.523872 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.523848 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbf29\" (UniqueName: \"kubernetes.io/projected/6f99a456-9f20-4d79-950e-a79b016f5043-kube-api-access-bbf29\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.523935 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.523893 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-credential-socket\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.524017 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.523981 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f99a456-9f20-4d79-950e-a79b016f5043-istiod-ca-cert\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.524150 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.524030 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-istio-data\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.555344 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.555313 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:50.594087 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.594057 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:50.624745 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.624713 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbf29\" (UniqueName: \"kubernetes.io/projected/6f99a456-9f20-4d79-950e-a79b016f5043-kube-api-access-bbf29\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.625223 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.624778 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-credential-socket\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.625223 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.624819 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f99a456-9f20-4d79-950e-a79b016f5043-istiod-ca-cert\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.625223 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.624852 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-istio-data\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.625223 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.624896 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-workload-certs\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.625223 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.624927 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f99a456-9f20-4d79-950e-a79b016f5043-istio-token\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.625223 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.624997 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f99a456-9f20-4d79-950e-a79b016f5043-istio-podinfo\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.625223 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.625045 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-workload-socket\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.625223 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.625102 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-istio-envoy\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.625495 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.625236 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-credential-socket\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.625535 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.625496 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-workload-certs\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.625535 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.625513 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-workload-socket\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.625672 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.625649 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f99a456-9f20-4d79-950e-a79b016f5043-istiod-ca-cert\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.625833 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.625812 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-istio-data\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.628085 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.628068 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f99a456-9f20-4d79-950e-a79b016f5043-istio-podinfo\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.628224 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.628210 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-istio-envoy\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.644401 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.644345 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbf29\" (UniqueName: \"kubernetes.io/projected/6f99a456-9f20-4d79-950e-a79b016f5043-kube-api-access-bbf29\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.644973 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.644950 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f99a456-9f20-4d79-950e-a79b016f5043-istio-token\") pod \"e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.696545 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.696509 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:50.725401 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.725372 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:50.796144 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.796114 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k"] Oct 30 00:55:50.800575 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.800557 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:50.815616 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.815590 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ba3d7b-openshift-default-dockercfg-6zwjk\"" Oct 30 00:55:50.820075 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.820053 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k"] Oct 30 00:55:50.822664 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.822628 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:50.870542 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.870509 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv"] Oct 30 00:55:50.873597 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:55:50.873570 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78eeb03d_9184_4a3d_af85_b8391731e828.slice/crio-8191207da3a16a01ad6652c2c6160fb98000615332c24e3a25512d24931de1fb WatchSource:0}: Error finding container 8191207da3a16a01ad6652c2c6160fb98000615332c24e3a25512d24931de1fb: Status 404 returned error can't find the container with id 8191207da3a16a01ad6652c2c6160fb98000615332c24e3a25512d24931de1fb Oct 30 00:55:50.875770 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.875726 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:55:50.875845 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.875821 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:55:50.875885 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.875850 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:55:50.911210 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.911188 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:50.928546 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.928414 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-workload-certs\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:50.928546 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.928471 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-workload-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:50.928546 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.928503 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-credential-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:50.928546 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.928527 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-data\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:50.929208 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.928564 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-podinfo\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:50.929208 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.928603 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-token\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:50.929208 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.928645 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce797d22-c33f-4853-8c58-e014cd26bf5b-istiod-ca-cert\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:50.929208 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.928835 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-envoy\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:50.929208 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:50.928905 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h4gc\" (UniqueName: \"kubernetes.io/projected/ce797d22-c33f-4853-8c58-e014cd26bf5b-kube-api-access-4h4gc\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:51.007309 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.007238 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:51.007917 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.007890 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt"] Oct 30 00:55:51.009456 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:55:51.009435 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f99a456_9f20_4d79_950e_a79b016f5043.slice/crio-6efc9e01e06e01a46c6288ee62b5035bfa812346b623ba40dbf420d721c89722 WatchSource:0}: Error finding container 6efc9e01e06e01a46c6288ee62b5035bfa812346b623ba40dbf420d721c89722: Status 404 returned error can't find the container with id 6efc9e01e06e01a46c6288ee62b5035bfa812346b623ba40dbf420d721c89722 Oct 30 00:55:51.012220 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.012187 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:55:51.012312 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.012271 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:55:51.012376 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.012315 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:55:51.029406 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.029380 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce797d22-c33f-4853-8c58-e014cd26bf5b-istiod-ca-cert\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:51.029515 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.029437 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-envoy\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:51.029515 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.029510 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h4gc\" (UniqueName: \"kubernetes.io/projected/ce797d22-c33f-4853-8c58-e014cd26bf5b-kube-api-access-4h4gc\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:51.029627 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.029586 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-workload-certs\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:51.029627 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.029615 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-workload-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:51.029732 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.029648 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-credential-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:51.029732 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.029677 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-data\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:51.029732 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.029720 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-podinfo\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:51.029906 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.029776 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-token\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:51.030334 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.030161 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-workload-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:51.030334 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.030190 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce797d22-c33f-4853-8c58-e014cd26bf5b-istiod-ca-cert\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:51.030334 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.030284 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-data\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:51.030518 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.030383 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-workload-certs\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:51.030576 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.030554 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-credential-socket\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:51.031970 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.031938 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-envoy\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:51.032345 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.032329 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-podinfo\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:51.055292 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.055272 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-token\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:51.055592 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.055570 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h4gc\" (UniqueName: \"kubernetes.io/projected/ce797d22-c33f-4853-8c58-e014cd26bf5b-kube-api-access-4h4gc\") pod \"e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:51.097277 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.097243 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:51.119833 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.119751 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:51.207541 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.199490 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:51.426282 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.426205 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k"] Oct 30 00:55:51.430484 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:55:51.430426 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce797d22_c33f_4853_8c58_e014cd26bf5b.slice/crio-9d3ae85541b8b065d6acb019f3ea664ee473292e7c44767f9e5b096802f4ccab WatchSource:0}: Error finding container 9d3ae85541b8b065d6acb019f3ea664ee473292e7c44767f9e5b096802f4ccab: Status 404 returned error can't find the container with id 9d3ae85541b8b065d6acb019f3ea664ee473292e7c44767f9e5b096802f4ccab Oct 30 00:55:51.433161 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.433128 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:55:51.433245 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.433194 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:55:51.433245 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.433222 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:55:51.592521 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:55:51.592481 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:55:51.797854 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.797815 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" event={"ID":"78eeb03d-9184-4a3d-af85-b8391731e828","Type":"ContainerStarted","Data":"596071fa4f369aa0913c47126c60790e71a4f450fc817aab4668ab7904f8522d"} Oct 30 00:55:51.798299 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.797880 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" event={"ID":"78eeb03d-9184-4a3d-af85-b8391731e828","Type":"ContainerStarted","Data":"8191207da3a16a01ad6652c2c6160fb98000615332c24e3a25512d24931de1fb"} Oct 30 00:55:51.799438 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.799409 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" event={"ID":"6f99a456-9f20-4d79-950e-a79b016f5043","Type":"ContainerStarted","Data":"9405e2ce69ba988f41b1f3156dd2913008d77daf182dd0b6fa89adc2f23f58ba"} Oct 30 00:55:51.799570 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.799457 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" event={"ID":"6f99a456-9f20-4d79-950e-a79b016f5043","Type":"ContainerStarted","Data":"6efc9e01e06e01a46c6288ee62b5035bfa812346b623ba40dbf420d721c89722"} Oct 30 00:55:51.800830 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.800807 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" event={"ID":"ce797d22-c33f-4853-8c58-e014cd26bf5b","Type":"ContainerStarted","Data":"b6c96873414c39606d49caf52f7f34febe7ae307d4bc695f36d02bbf62dae394"} Oct 30 00:55:51.800936 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.800852 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" event={"ID":"ce797d22-c33f-4853-8c58-e014cd26bf5b","Type":"ContainerStarted","Data":"9d3ae85541b8b065d6acb019f3ea664ee473292e7c44767f9e5b096802f4ccab"} Oct 30 00:55:51.849191 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.849140 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" podStartSLOduration=1.849127847 podStartE2EDuration="1.849127847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:51.848209728 +0000 UTC m=+752.164078052" watchObservedRunningTime="2026-10-30 00:55:51.849127847 +0000 UTC m=+752.164996171" Oct 30 00:55:51.900421 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.900375 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" podStartSLOduration=1.900363134 podStartE2EDuration="1.900363134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:51.898278735 +0000 UTC m=+752.214147060" watchObservedRunningTime="2026-10-30 00:55:51.900363134 +0000 UTC m=+752.216231459" Oct 30 00:55:51.939991 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:51.939952 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" podStartSLOduration=1.939937401 podStartE2EDuration="1.939937401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:51.938055463 +0000 UTC m=+752.253923789" watchObservedRunningTime="2026-10-30 00:55:51.939937401 +0000 UTC m=+752.255805726" Oct 30 00:55:52.120078 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:52.119997 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:52.125034 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:52.125009 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:52.555713 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:52.555675 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:52.560924 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:52.560899 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:52.696909 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:52.696872 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:52.702283 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:52.702257 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:52.806622 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:52.806426 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:52.806622 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:52.806478 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:52.806622 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:52.806516 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:52.807165 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:52.807124 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:55:52.807220 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:52.807182 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:55:52.807275 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:52.807242 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:55:53.788602 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:53.788560 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:53.891391 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:53.891342 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:53.992011 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:53.991976 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:54.238056 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:54.238026 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:54.339562 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:54.339524 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:54.440085 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:54.440053 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:54.560636 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:54.560539 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:54.670479 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:54.670440 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:54.777479 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:54.777443 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:54.903195 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:54.903095 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:55.003428 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:55.003399 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:55.476074 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:55.476005 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:55.591257 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:55.591215 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:55.695972 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:55.695924 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:55.776345 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:55.776274 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb"] Oct 30 00:55:55.779938 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:55.779919 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb" Oct 30 00:55:55.812312 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:55.812292 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-dca38d\"" Oct 30 00:55:55.824569 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:55.824541 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb"] Oct 30 00:55:55.847668 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:55.847641 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm"] Oct 30 00:55:55.851864 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:55.851846 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm" Oct 30 00:55:55.870561 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:55.870534 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm"] Oct 30 00:55:55.873867 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:55.873844 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcwvs\" (UniqueName: \"kubernetes.io/projected/7f309af2-1f23-4295-8187-129275259d8a-kube-api-access-bcwvs\") pod \"payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb\" (UID: \"7f309af2-1f23-4295-8187-129275259d8a\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb" Oct 30 00:55:55.874014 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:55.873990 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f309af2-1f23-4295-8187-129275259d8a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb\" (UID: \"7f309af2-1f23-4295-8187-129275259d8a\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb" Oct 30 00:55:55.880737 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:55.880716 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-dca38d-dockercfg-gq5cp\"" Oct 30 00:55:55.913019 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:55.912987 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:55.975440 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:55.975392 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c19f786-50e8-4dee-97ae-c4445035f930-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm\" (UID: \"5c19f786-50e8-4dee-97ae-c4445035f930\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm" Oct 30 00:55:55.975610 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:55.975469 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcwvs\" (UniqueName: \"kubernetes.io/projected/7f309af2-1f23-4295-8187-129275259d8a-kube-api-access-bcwvs\") pod \"payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb\" (UID: \"7f309af2-1f23-4295-8187-129275259d8a\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb" Oct 30 00:55:55.975610 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:55.975570 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5gbc\" (UniqueName: \"kubernetes.io/projected/5c19f786-50e8-4dee-97ae-c4445035f930-kube-api-access-p5gbc\") pod \"payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm\" (UID: \"5c19f786-50e8-4dee-97ae-c4445035f930\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm" Oct 30 00:55:55.975740 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:55.975611 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f309af2-1f23-4295-8187-129275259d8a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb\" (UID: \"7f309af2-1f23-4295-8187-129275259d8a\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb" Oct 30 00:55:55.976571 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:55.976545 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f309af2-1f23-4295-8187-129275259d8a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb\" (UID: \"7f309af2-1f23-4295-8187-129275259d8a\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb" Oct 30 00:55:55.997560 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:55.997528 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcwvs\" (UniqueName: \"kubernetes.io/projected/7f309af2-1f23-4295-8187-129275259d8a-kube-api-access-bcwvs\") pod \"payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb\" (UID: \"7f309af2-1f23-4295-8187-129275259d8a\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb" Oct 30 00:55:56.076388 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:56.076292 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c19f786-50e8-4dee-97ae-c4445035f930-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm\" (UID: \"5c19f786-50e8-4dee-97ae-c4445035f930\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm" Oct 30 00:55:56.076388 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:56.076392 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5gbc\" (UniqueName: \"kubernetes.io/projected/5c19f786-50e8-4dee-97ae-c4445035f930-kube-api-access-p5gbc\") pod \"payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm\" (UID: \"5c19f786-50e8-4dee-97ae-c4445035f930\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm" Oct 30 00:55:56.077078 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:56.077058 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c19f786-50e8-4dee-97ae-c4445035f930-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm\" (UID: \"5c19f786-50e8-4dee-97ae-c4445035f930\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm" Oct 30 00:55:56.092962 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:56.092937 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb" Oct 30 00:55:56.096001 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:56.095979 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5gbc\" (UniqueName: \"kubernetes.io/projected/5c19f786-50e8-4dee-97ae-c4445035f930-kube-api-access-p5gbc\") pod \"payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm\" (UID: \"5c19f786-50e8-4dee-97ae-c4445035f930\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm" Oct 30 00:55:56.165596 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:56.165567 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm" Oct 30 00:55:56.391828 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:56.391734 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb"] Oct 30 00:55:56.394682 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:55:56.394649 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f309af2_1f23_4295_8187_129275259d8a.slice/crio-8af764958954d12ee568c25be441a0ed8952eea3ffe786eb8e37403179ba69c6 WatchSource:0}: Error finding container 8af764958954d12ee568c25be441a0ed8952eea3ffe786eb8e37403179ba69c6: Status 404 returned error can't find the container with id 8af764958954d12ee568c25be441a0ed8952eea3ffe786eb8e37403179ba69c6 Oct 30 00:55:56.473041 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:56.473012 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm"] Oct 30 00:55:56.475305 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:55:56.475279 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c19f786_50e8_4dee_97ae_c4445035f930.slice/crio-8827f2bf0bd025094fb6543d301a8b35694a7e97163dc3ab93625c9d24620694 WatchSource:0}: Error finding container 8827f2bf0bd025094fb6543d301a8b35694a7e97163dc3ab93625c9d24620694: Status 404 returned error can't find the container with id 8827f2bf0bd025094fb6543d301a8b35694a7e97163dc3ab93625c9d24620694 Oct 30 00:55:56.826542 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:56.826500 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb" event={"ID":"7f309af2-1f23-4295-8187-129275259d8a","Type":"ContainerStarted","Data":"981208c4e05bf33db5ad768118b71b789f8794874cea061663aa65a16be50af5"} Oct 30 00:55:56.826752 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:56.826564 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb" event={"ID":"7f309af2-1f23-4295-8187-129275259d8a","Type":"ContainerStarted","Data":"8af764958954d12ee568c25be441a0ed8952eea3ffe786eb8e37403179ba69c6"} Oct 30 00:55:56.826752 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:56.826595 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb" Oct 30 00:55:56.828146 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:56.828121 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm" event={"ID":"5c19f786-50e8-4dee-97ae-c4445035f930","Type":"ContainerStarted","Data":"afbf1873ed4296546c649b91a42b600c578726cb07be0da8c37020648290bcdb"} Oct 30 00:55:56.828281 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:56.828161 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm" event={"ID":"5c19f786-50e8-4dee-97ae-c4445035f930","Type":"ContainerStarted","Data":"8827f2bf0bd025094fb6543d301a8b35694a7e97163dc3ab93625c9d24620694"} Oct 30 00:55:56.828281 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:56.828183 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm" Oct 30 00:55:56.866196 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:56.866139 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb" podStartSLOduration=1.8661234979999999 podStartE2EDuration="1.866123498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:56.864403434 +0000 UTC m=+757.180271760" watchObservedRunningTime="2026-10-30 00:55:56.866123498 +0000 UTC m=+757.181991823" Oct 30 00:55:56.905934 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:56.905885 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm" podStartSLOduration=1.90587085 podStartE2EDuration="1.90587085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:56.904356112 +0000 UTC m=+757.220224437" watchObservedRunningTime="2026-10-30 00:55:56.90587085 +0000 UTC m=+757.221739174" Oct 30 00:55:57.124137 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:57.124064 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz"] Oct 30 00:55:57.127794 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:57.127755 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz" Oct 30 00:55:57.158574 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:57.158543 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz"] Oct 30 00:55:57.203270 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:57.203238 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:57.214661 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:57.214638 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-c104b8-serving-cert\"" Oct 30 00:55:57.288355 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:57.288317 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz\" (UID: \"ec2520e7-a5f7-4127-ac4b-5bb40efc6309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz" Oct 30 00:55:57.288355 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:57.288357 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-metrics-certs\") pod \"maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz\" (UID: \"ec2520e7-a5f7-4127-ac4b-5bb40efc6309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz" Oct 30 00:55:57.288617 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:57.288443 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v2jk\" (UniqueName: \"kubernetes.io/projected/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-kube-api-access-2v2jk\") pod \"maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz\" (UID: \"ec2520e7-a5f7-4127-ac4b-5bb40efc6309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz" Oct 30 00:55:57.288617 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:57.288497 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-maas-api-tls\") pod \"maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz\" (UID: \"ec2520e7-a5f7-4127-ac4b-5bb40efc6309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz" Oct 30 00:55:57.303519 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:57.303494 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:57.389191 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:57.389114 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz\" (UID: \"ec2520e7-a5f7-4127-ac4b-5bb40efc6309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz" Oct 30 00:55:57.389191 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:57.389147 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-metrics-certs\") pod \"maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz\" (UID: \"ec2520e7-a5f7-4127-ac4b-5bb40efc6309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz" Oct 30 00:55:57.389191 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:57.389181 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v2jk\" (UniqueName: \"kubernetes.io/projected/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-kube-api-access-2v2jk\") pod \"maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz\" (UID: \"ec2520e7-a5f7-4127-ac4b-5bb40efc6309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz" Oct 30 00:55:57.389411 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:57.389214 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-maas-api-tls\") pod \"maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz\" (UID: \"ec2520e7-a5f7-4127-ac4b-5bb40efc6309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz" Oct 30 00:55:57.389859 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:57.389822 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz\" (UID: \"ec2520e7-a5f7-4127-ac4b-5bb40efc6309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz" Oct 30 00:55:57.391696 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:57.391673 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-metrics-certs\") pod \"maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz\" (UID: \"ec2520e7-a5f7-4127-ac4b-5bb40efc6309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz" Oct 30 00:55:57.391907 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:57.391889 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-maas-api-tls\") pod \"maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz\" (UID: \"ec2520e7-a5f7-4127-ac4b-5bb40efc6309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz" Oct 30 00:55:57.408533 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:57.408510 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v2jk\" (UniqueName: \"kubernetes.io/projected/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-kube-api-access-2v2jk\") pod \"maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz\" (UID: \"ec2520e7-a5f7-4127-ac4b-5bb40efc6309\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz" Oct 30 00:55:57.440545 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:57.440521 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz" Oct 30 00:55:57.505610 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:57.505580 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:57.595615 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:57.595584 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:57.738976 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:57.738940 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz"] Oct 30 00:55:57.743270 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:55:57.743236 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2520e7_a5f7_4127_ac4b_5bb40efc6309.slice/crio-78db8494920cb95221a6932de507bab57822a21b6172fcb6bf8fd28cd14cfc67 WatchSource:0}: Error finding container 78db8494920cb95221a6932de507bab57822a21b6172fcb6bf8fd28cd14cfc67: Status 404 returned error can't find the container with id 78db8494920cb95221a6932de507bab57822a21b6172fcb6bf8fd28cd14cfc67 Oct 30 00:55:57.744944 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:57.744913 5300 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 00:55:57.833339 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:57.833297 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz" event={"ID":"ec2520e7-a5f7-4127-ac4b-5bb40efc6309","Type":"ContainerStarted","Data":"78db8494920cb95221a6932de507bab57822a21b6172fcb6bf8fd28cd14cfc67"} Oct 30 00:55:58.326712 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:58.326672 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:58.428224 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:58.428148 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:58.841239 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:58.841206 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz" event={"ID":"ec2520e7-a5f7-4127-ac4b-5bb40efc6309","Type":"ContainerStarted","Data":"4a4e04ea7a262a8ee834fbce9222ef83088803fc9313d6e12fdd441f7fc36d24"} Oct 30 00:55:58.841414 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:58.841269 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz" Oct 30 00:55:59.016418 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:59.016355 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz" podStartSLOduration=1.63320844 podStartE2EDuration="2.016334039s" totalImagesPullingTime="383.125599ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:58.890818486 +0000 UTC m=+759.206686811" watchObservedRunningTime="2026-10-30 00:55:59.016334039 +0000 UTC m=+759.332202365" Oct 30 00:55:59.017044 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:59.017025 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls"] Oct 30 00:55:59.020737 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:59.020721 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls" Oct 30 00:55:59.036749 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:59.036720 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-ba3d7b-serving-cert\"" Oct 30 00:55:59.038777 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:59.038731 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls"] Oct 30 00:55:59.101558 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:59.101479 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:59.203863 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:59.203828 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:55:59.208020 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:59.207995 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xvfv\" (UniqueName: \"kubernetes.io/projected/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-kube-api-access-4xvfv\") pod \"maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls\" (UID: \"151eb97e-bb5b-44ca-89c9-07025f5cb2d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls" Oct 30 00:55:59.208164 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:59.208044 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls\" (UID: \"151eb97e-bb5b-44ca-89c9-07025f5cb2d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls" Oct 30 00:55:59.208164 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:59.208112 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls\" (UID: \"151eb97e-bb5b-44ca-89c9-07025f5cb2d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls" Oct 30 00:55:59.208247 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:59.208206 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-metrics-certs\") pod \"maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls\" (UID: \"151eb97e-bb5b-44ca-89c9-07025f5cb2d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls" Oct 30 00:55:59.308946 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:59.308906 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls\" (UID: \"151eb97e-bb5b-44ca-89c9-07025f5cb2d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls" Oct 30 00:55:59.309162 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:59.309017 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-metrics-certs\") pod \"maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls\" (UID: \"151eb97e-bb5b-44ca-89c9-07025f5cb2d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls" Oct 30 00:55:59.309162 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:59.309042 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xvfv\" (UniqueName: \"kubernetes.io/projected/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-kube-api-access-4xvfv\") pod \"maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls\" (UID: \"151eb97e-bb5b-44ca-89c9-07025f5cb2d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls" Oct 30 00:55:59.309162 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:59.309073 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls\" (UID: \"151eb97e-bb5b-44ca-89c9-07025f5cb2d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls" Oct 30 00:55:59.309681 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:59.309655 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls\" (UID: \"151eb97e-bb5b-44ca-89c9-07025f5cb2d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls" Oct 30 00:55:59.311569 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:59.311538 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls\" (UID: \"151eb97e-bb5b-44ca-89c9-07025f5cb2d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls" Oct 30 00:55:59.311659 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:59.311579 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-metrics-certs\") pod \"maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls\" (UID: \"151eb97e-bb5b-44ca-89c9-07025f5cb2d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls" Oct 30 00:55:59.328025 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:59.327991 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xvfv\" (UniqueName: \"kubernetes.io/projected/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-kube-api-access-4xvfv\") pod \"maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls\" (UID: \"151eb97e-bb5b-44ca-89c9-07025f5cb2d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls" Oct 30 00:55:59.332954 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:59.332929 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls" Oct 30 00:55:59.629263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:59.629220 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls"] Oct 30 00:55:59.634394 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:55:59.634334 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod151eb97e_bb5b_44ca_89c9_07025f5cb2d6.slice/crio-9a30bcc23d9ca15ef75b36a7d9e1e86452157626ddc96c13f4fb5486259abc04 WatchSource:0}: Error finding container 9a30bcc23d9ca15ef75b36a7d9e1e86452157626ddc96c13f4fb5486259abc04: Status 404 returned error can't find the container with id 9a30bcc23d9ca15ef75b36a7d9e1e86452157626ddc96c13f4fb5486259abc04 Oct 30 00:55:59.846584 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:59.846549 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls" event={"ID":"151eb97e-bb5b-44ca-89c9-07025f5cb2d6","Type":"ContainerStarted","Data":"9a30bcc23d9ca15ef75b36a7d9e1e86452157626ddc96c13f4fb5486259abc04"} Oct 30 00:55:59.905484 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:55:59.905405 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:00.004775 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:00.004741 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:00.729843 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:00.729807 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:00.824242 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:00.824209 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:00.852978 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:00.852943 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls" event={"ID":"151eb97e-bb5b-44ca-89c9-07025f5cb2d6","Type":"ContainerStarted","Data":"3b21149e00670135dbded8b7806ddc1d41ec94555a0c0086284737b1be18cdd7"} Oct 30 00:56:00.853142 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:00.853048 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls" Oct 30 00:56:00.915382 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:00.915332 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls" podStartSLOduration=2.593819011 podStartE2EDuration="2.915317544s" totalImagesPullingTime="321.498533ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:00.900425543 +0000 UTC m=+761.216293869" watchObservedRunningTime="2026-10-30 00:56:00.915317544 +0000 UTC m=+761.231185870" Oct 30 00:56:00.924790 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:00.924747 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:00.926264 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:00.926241 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw"] Oct 30 00:56:00.929911 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:00.929895 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw" Oct 30 00:56:00.947433 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:00.947411 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw"] Oct 30 00:56:00.949159 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:00.949135 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-fd3281\"" Oct 30 00:56:00.949307 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:00.949288 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-fd3281-dockercfg-wnjsj\"" Oct 30 00:56:01.014640 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:01.014567 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl"] Oct 30 00:56:01.018283 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:01.018260 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl" Oct 30 00:56:01.024346 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:01.024323 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fd8605b-ff02-4868-a951-8fe172766e88-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw\" (UID: \"1fd8605b-ff02-4868-a951-8fe172766e88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw" Oct 30 00:56:01.024444 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:01.024373 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgcp9\" (UniqueName: \"kubernetes.io/projected/d3fb7228-7fdb-4db8-a878-d6f902cbc170-kube-api-access-wgcp9\") pod \"payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl\" (UID: \"d3fb7228-7fdb-4db8-a878-d6f902cbc170\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl" Oct 30 00:56:01.024598 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:01.024576 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3fb7228-7fdb-4db8-a878-d6f902cbc170-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl\" (UID: \"d3fb7228-7fdb-4db8-a878-d6f902cbc170\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl" Oct 30 00:56:01.024670 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:01.024631 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5hxt\" (UniqueName: \"kubernetes.io/projected/1fd8605b-ff02-4868-a951-8fe172766e88-kube-api-access-s5hxt\") pod \"payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw\" (UID: \"1fd8605b-ff02-4868-a951-8fe172766e88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw" Oct 30 00:56:01.026284 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:01.026263 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:01.037465 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:01.037441 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl"] Oct 30 00:56:01.125556 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:01.125510 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3fb7228-7fdb-4db8-a878-d6f902cbc170-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl\" (UID: \"d3fb7228-7fdb-4db8-a878-d6f902cbc170\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl" Oct 30 00:56:01.125795 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:01.125579 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5hxt\" (UniqueName: \"kubernetes.io/projected/1fd8605b-ff02-4868-a951-8fe172766e88-kube-api-access-s5hxt\") pod \"payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw\" (UID: \"1fd8605b-ff02-4868-a951-8fe172766e88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw" Oct 30 00:56:01.125795 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:01.125649 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fd8605b-ff02-4868-a951-8fe172766e88-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw\" (UID: \"1fd8605b-ff02-4868-a951-8fe172766e88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw" Oct 30 00:56:01.126087 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:01.126053 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgcp9\" (UniqueName: \"kubernetes.io/projected/d3fb7228-7fdb-4db8-a878-d6f902cbc170-kube-api-access-wgcp9\") pod \"payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl\" (UID: \"d3fb7228-7fdb-4db8-a878-d6f902cbc170\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl" Oct 30 00:56:01.126434 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:01.126412 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3fb7228-7fdb-4db8-a878-d6f902cbc170-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl\" (UID: \"d3fb7228-7fdb-4db8-a878-d6f902cbc170\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl" Oct 30 00:56:01.138045 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:01.138012 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fd8605b-ff02-4868-a951-8fe172766e88-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw\" (UID: \"1fd8605b-ff02-4868-a951-8fe172766e88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw" Oct 30 00:56:01.156782 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:01.153915 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5hxt\" (UniqueName: \"kubernetes.io/projected/1fd8605b-ff02-4868-a951-8fe172766e88-kube-api-access-s5hxt\") pod \"payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw\" (UID: \"1fd8605b-ff02-4868-a951-8fe172766e88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw" Oct 30 00:56:01.163323 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:01.163280 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgcp9\" (UniqueName: \"kubernetes.io/projected/d3fb7228-7fdb-4db8-a878-d6f902cbc170-kube-api-access-wgcp9\") pod \"payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl\" (UID: \"d3fb7228-7fdb-4db8-a878-d6f902cbc170\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl" Oct 30 00:56:01.250785 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:01.247987 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw" Oct 30 00:56:01.333190 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:01.332152 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl" Oct 30 00:56:01.578852 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:01.578754 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw"] Oct 30 00:56:01.582998 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:56:01.582971 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd8605b_ff02_4868_a951_8fe172766e88.slice/crio-68890c48fffe8b57826d3c21f42743f15ece255bd70b386b32306c428258303d WatchSource:0}: Error finding container 68890c48fffe8b57826d3c21f42743f15ece255bd70b386b32306c428258303d: Status 404 returned error can't find the container with id 68890c48fffe8b57826d3c21f42743f15ece255bd70b386b32306c428258303d Oct 30 00:56:01.666742 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:01.666718 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl"] Oct 30 00:56:01.669257 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:56:01.669190 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3fb7228_7fdb_4db8_a878_d6f902cbc170.slice/crio-9a966786a238c6db3f8108513936cb369c2cca3862acb88c9f32e4d67d677440 WatchSource:0}: Error finding container 9a966786a238c6db3f8108513936cb369c2cca3862acb88c9f32e4d67d677440: Status 404 returned error can't find the container with id 9a966786a238c6db3f8108513936cb369c2cca3862acb88c9f32e4d67d677440 Oct 30 00:56:01.859837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:01.859727 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl" event={"ID":"d3fb7228-7fdb-4db8-a878-d6f902cbc170","Type":"ContainerStarted","Data":"5af672de892d6565734d950bcf7e7423cdd4122c455ba611690e459ab2e35941"} Oct 30 00:56:01.859837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:01.859811 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl" event={"ID":"d3fb7228-7fdb-4db8-a878-d6f902cbc170","Type":"ContainerStarted","Data":"9a966786a238c6db3f8108513936cb369c2cca3862acb88c9f32e4d67d677440"} Oct 30 00:56:01.860302 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:01.859851 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl" Oct 30 00:56:01.861479 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:01.861452 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw" event={"ID":"1fd8605b-ff02-4868-a951-8fe172766e88","Type":"ContainerStarted","Data":"a0e810e2c3bfc7ea003835e85ce6cdcae08cf85bb01a4d72a4df1df7e5b1afc3"} Oct 30 00:56:01.861592 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:01.861498 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw" event={"ID":"1fd8605b-ff02-4868-a951-8fe172766e88","Type":"ContainerStarted","Data":"68890c48fffe8b57826d3c21f42743f15ece255bd70b386b32306c428258303d"} Oct 30 00:56:01.897811 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:01.897703 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl" podStartSLOduration=1.897684989 podStartE2EDuration="1.897684989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:01.896384762 +0000 UTC m=+762.212253087" watchObservedRunningTime="2026-10-30 00:56:01.897684989 +0000 UTC m=+762.213553310" Oct 30 00:56:01.934949 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:01.934598 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw" podStartSLOduration=1.934581234 podStartE2EDuration="1.934581234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:01.933538079 +0000 UTC m=+762.249406398" watchObservedRunningTime="2026-10-30 00:56:01.934581234 +0000 UTC m=+762.250449563" Oct 30 00:56:02.867501 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:02.867460 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw" Oct 30 00:56:02.911526 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:02.911495 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:02.989608 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:02.989569 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:03.101481 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:03.101445 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:03.201794 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:03.201682 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:03.302116 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:03.302075 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:03.317077 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:03.317044 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9"] Oct 30 00:56:03.345375 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:03.345078 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9"] Oct 30 00:56:03.345375 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:03.345198 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9" Oct 30 00:56:03.356553 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:03.356523 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5rjn\" (UniqueName: \"kubernetes.io/projected/8fd0d5d9-adcd-4d24-8916-379be6b844e9-kube-api-access-w5rjn\") pod \"payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9\" (UID: \"8fd0d5d9-adcd-4d24-8916-379be6b844e9\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9" Oct 30 00:56:03.356709 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:03.356584 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fd0d5d9-adcd-4d24-8916-379be6b844e9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9\" (UID: \"8fd0d5d9-adcd-4d24-8916-379be6b844e9\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9" Oct 30 00:56:03.362236 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:03.362212 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-c4847c\"" Oct 30 00:56:03.374443 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:03.374424 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-c4847c-dockercfg-5qdcl\"" Oct 30 00:56:03.457384 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:03.457292 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5rjn\" (UniqueName: \"kubernetes.io/projected/8fd0d5d9-adcd-4d24-8916-379be6b844e9-kube-api-access-w5rjn\") pod \"payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9\" (UID: \"8fd0d5d9-adcd-4d24-8916-379be6b844e9\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9" Oct 30 00:56:03.457384 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:03.457346 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fd0d5d9-adcd-4d24-8916-379be6b844e9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9\" (UID: \"8fd0d5d9-adcd-4d24-8916-379be6b844e9\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9" Oct 30 00:56:03.458407 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:03.458209 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fd0d5d9-adcd-4d24-8916-379be6b844e9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9\" (UID: \"8fd0d5d9-adcd-4d24-8916-379be6b844e9\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9" Oct 30 00:56:03.479099 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:03.478943 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5rjn\" (UniqueName: \"kubernetes.io/projected/8fd0d5d9-adcd-4d24-8916-379be6b844e9-kube-api-access-w5rjn\") pod \"payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9\" (UID: \"8fd0d5d9-adcd-4d24-8916-379be6b844e9\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9" Oct 30 00:56:03.662349 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:03.662312 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9" Oct 30 00:56:03.955567 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:03.955531 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9"] Oct 30 00:56:03.959134 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:56:03.959104 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd0d5d9_adcd_4d24_8916_379be6b844e9.slice/crio-3f606b19a4b2ef983e7fa00d1c3488b0372ff3d23aac1dc195556f26dd314b81 WatchSource:0}: Error finding container 3f606b19a4b2ef983e7fa00d1c3488b0372ff3d23aac1dc195556f26dd314b81: Status 404 returned error can't find the container with id 3f606b19a4b2ef983e7fa00d1c3488b0372ff3d23aac1dc195556f26dd314b81 Oct 30 00:56:04.064289 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:04.064241 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:04.162534 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:04.162504 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:04.645625 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:04.645592 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq"] Oct 30 00:56:04.649359 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:04.649335 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq" Oct 30 00:56:04.668888 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:04.668858 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2839a72-aea8-412d-b105-cf819e2e497a-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq\" (UID: \"c2839a72-aea8-412d-b105-cf819e2e497a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq" Oct 30 00:56:04.669020 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:04.668986 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2839a72-aea8-412d-b105-cf819e2e497a-metrics-certs\") pod \"maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq\" (UID: \"c2839a72-aea8-412d-b105-cf819e2e497a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq" Oct 30 00:56:04.669091 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:04.669058 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvwrb\" (UniqueName: \"kubernetes.io/projected/c2839a72-aea8-412d-b105-cf819e2e497a-kube-api-access-mvwrb\") pod \"maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq\" (UID: \"c2839a72-aea8-412d-b105-cf819e2e497a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq" Oct 30 00:56:04.669140 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:04.669123 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2839a72-aea8-412d-b105-cf819e2e497a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq\" (UID: \"c2839a72-aea8-412d-b105-cf819e2e497a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq" Oct 30 00:56:04.676650 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:04.676411 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-d827b7-serving-cert\"" Oct 30 00:56:04.679412 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:04.679382 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq"] Oct 30 00:56:04.716047 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:04.716014 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:04.769682 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:04.769647 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2839a72-aea8-412d-b105-cf819e2e497a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq\" (UID: \"c2839a72-aea8-412d-b105-cf819e2e497a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq" Oct 30 00:56:04.769878 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:04.769755 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2839a72-aea8-412d-b105-cf819e2e497a-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq\" (UID: \"c2839a72-aea8-412d-b105-cf819e2e497a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq" Oct 30 00:56:04.769878 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:56:04.769819 5300 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-serving-cert: secret "maas-api-e2e-worker-w4-d827b7-serving-cert" not found Oct 30 00:56:04.769878 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:04.769847 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2839a72-aea8-412d-b105-cf819e2e497a-metrics-certs\") pod \"maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq\" (UID: \"c2839a72-aea8-412d-b105-cf819e2e497a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq" Oct 30 00:56:04.770031 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:56:04.769889 5300 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2839a72-aea8-412d-b105-cf819e2e497a-maas-api-tls podName:c2839a72-aea8-412d-b105-cf819e2e497a nodeName:}" failed. No retries permitted until 2026-10-30 00:56:05.26986681 +0000 UTC m=+765.585735115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c2839a72-aea8-412d-b105-cf819e2e497a-maas-api-tls") pod "maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq" (UID: "c2839a72-aea8-412d-b105-cf819e2e497a") : secret "maas-api-e2e-worker-w4-d827b7-serving-cert" not found Oct 30 00:56:04.770031 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:04.769934 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvwrb\" (UniqueName: \"kubernetes.io/projected/c2839a72-aea8-412d-b105-cf819e2e497a-kube-api-access-mvwrb\") pod \"maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq\" (UID: \"c2839a72-aea8-412d-b105-cf819e2e497a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq" Oct 30 00:56:04.770813 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:04.770793 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2839a72-aea8-412d-b105-cf819e2e497a-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq\" (UID: \"c2839a72-aea8-412d-b105-cf819e2e497a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq" Oct 30 00:56:04.772225 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:04.772207 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2839a72-aea8-412d-b105-cf819e2e497a-metrics-certs\") pod \"maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq\" (UID: \"c2839a72-aea8-412d-b105-cf819e2e497a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq" Oct 30 00:56:04.789392 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:04.789363 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvwrb\" (UniqueName: \"kubernetes.io/projected/c2839a72-aea8-412d-b105-cf819e2e497a-kube-api-access-mvwrb\") pod \"maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq\" (UID: \"c2839a72-aea8-412d-b105-cf819e2e497a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq" Oct 30 00:56:04.800168 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:04.800141 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:04.856840 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:04.856812 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz" Oct 30 00:56:04.881262 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:04.881226 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9" event={"ID":"8fd0d5d9-adcd-4d24-8916-379be6b844e9","Type":"ContainerStarted","Data":"e9a5a8c0cb1cc9df7107e30dbb93dd8890e435a01e36f0c4ee7f5c82b5c69c75"} Oct 30 00:56:04.881434 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:04.881287 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9" event={"ID":"8fd0d5d9-adcd-4d24-8916-379be6b844e9","Type":"ContainerStarted","Data":"3f606b19a4b2ef983e7fa00d1c3488b0372ff3d23aac1dc195556f26dd314b81"} Oct 30 00:56:04.881434 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:04.881374 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9" Oct 30 00:56:04.907877 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:04.907799 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:04.947330 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:04.945482 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9" podStartSLOduration=1.945459493 podStartE2EDuration="1.945459493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:04.942189942 +0000 UTC m=+765.258058268" watchObservedRunningTime="2026-10-30 00:56:04.945459493 +0000 UTC m=+765.261327819" Oct 30 00:56:05.275033 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:05.274992 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2839a72-aea8-412d-b105-cf819e2e497a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq\" (UID: \"c2839a72-aea8-412d-b105-cf819e2e497a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq" Oct 30 00:56:05.277407 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:05.277379 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2839a72-aea8-412d-b105-cf819e2e497a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq\" (UID: \"c2839a72-aea8-412d-b105-cf819e2e497a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq" Oct 30 00:56:05.562466 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:05.562350 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq" Oct 30 00:56:05.870603 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:05.870519 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq"] Oct 30 00:56:05.873672 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:56:05.873637 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2839a72_aea8_412d_b105_cf819e2e497a.slice/crio-8b9ec5e6f6c1e12058058b4b0b46bfc2f350d7658a40068d00c137c71d4b12f9 WatchSource:0}: Error finding container 8b9ec5e6f6c1e12058058b4b0b46bfc2f350d7658a40068d00c137c71d4b12f9: Status 404 returned error can't find the container with id 8b9ec5e6f6c1e12058058b4b0b46bfc2f350d7658a40068d00c137c71d4b12f9 Oct 30 00:56:05.887713 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:05.887687 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq" event={"ID":"c2839a72-aea8-412d-b105-cf819e2e497a","Type":"ContainerStarted","Data":"8b9ec5e6f6c1e12058058b4b0b46bfc2f350d7658a40068d00c137c71d4b12f9"} Oct 30 00:56:06.601554 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:06.601467 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:06.868011 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:06.867935 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls" Oct 30 00:56:06.895700 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:06.895670 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq" event={"ID":"c2839a72-aea8-412d-b105-cf819e2e497a","Type":"ContainerStarted","Data":"db74a89c8147cee04cb6ae268efdb792ab6936ad1a2051ba37f0d761bcf4570d"} Oct 30 00:56:06.895866 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:06.895779 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq" Oct 30 00:56:06.988327 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:06.988278 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq" podStartSLOduration=2.5942448799999998 podStartE2EDuration="2.988264053s" totalImagesPullingTime="394.019173ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:06.988144115 +0000 UTC m=+767.304012441" watchObservedRunningTime="2026-10-30 00:56:06.988264053 +0000 UTC m=+767.304132377" Oct 30 00:56:07.846199 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:07.846153 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb" Oct 30 00:56:07.847103 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:07.847078 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm" Oct 30 00:56:08.467021 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:08.466984 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:10.205833 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:10.205801 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:10.306554 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:10.306520 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:10.431094 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:10.431048 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:12.349188 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.349148 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr"] Oct 30 00:56:12.354072 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.354051 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr" Oct 30 00:56:12.369339 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.369314 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr"] Oct 30 00:56:12.379478 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.379452 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-fd3281-serving-cert\"" Oct 30 00:56:12.444325 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.444232 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:12.488194 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.488155 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nmqx\" (UniqueName: \"kubernetes.io/projected/fdd0a075-d5cf-4982-9623-c5432a308de6-kube-api-access-4nmqx\") pod \"maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr\" (UID: \"fdd0a075-d5cf-4982-9623-c5432a308de6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr" Oct 30 00:56:12.488361 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.488213 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdd0a075-d5cf-4982-9623-c5432a308de6-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr\" (UID: \"fdd0a075-d5cf-4982-9623-c5432a308de6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr" Oct 30 00:56:12.488361 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.488344 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdd0a075-d5cf-4982-9623-c5432a308de6-maas-api-tls\") pod \"maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr\" (UID: \"fdd0a075-d5cf-4982-9623-c5432a308de6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr" Oct 30 00:56:12.488446 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.488399 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd0a075-d5cf-4982-9623-c5432a308de6-metrics-certs\") pod \"maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr\" (UID: \"fdd0a075-d5cf-4982-9623-c5432a308de6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr" Oct 30 00:56:12.542990 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.542902 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:12.590054 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.590011 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nmqx\" (UniqueName: \"kubernetes.io/projected/fdd0a075-d5cf-4982-9623-c5432a308de6-kube-api-access-4nmqx\") pod \"maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr\" (UID: \"fdd0a075-d5cf-4982-9623-c5432a308de6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr" Oct 30 00:56:12.590054 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.590059 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdd0a075-d5cf-4982-9623-c5432a308de6-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr\" (UID: \"fdd0a075-d5cf-4982-9623-c5432a308de6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr" Oct 30 00:56:12.590322 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.590128 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdd0a075-d5cf-4982-9623-c5432a308de6-maas-api-tls\") pod \"maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr\" (UID: \"fdd0a075-d5cf-4982-9623-c5432a308de6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr" Oct 30 00:56:12.590322 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.590180 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd0a075-d5cf-4982-9623-c5432a308de6-metrics-certs\") pod \"maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr\" (UID: \"fdd0a075-d5cf-4982-9623-c5432a308de6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr" Oct 30 00:56:12.590820 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.590797 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdd0a075-d5cf-4982-9623-c5432a308de6-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr\" (UID: \"fdd0a075-d5cf-4982-9623-c5432a308de6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr" Oct 30 00:56:12.592560 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.592539 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd0a075-d5cf-4982-9623-c5432a308de6-metrics-certs\") pod \"maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr\" (UID: \"fdd0a075-d5cf-4982-9623-c5432a308de6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr" Oct 30 00:56:12.592664 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.592591 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdd0a075-d5cf-4982-9623-c5432a308de6-maas-api-tls\") pod \"maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr\" (UID: \"fdd0a075-d5cf-4982-9623-c5432a308de6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr" Oct 30 00:56:12.609082 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.609056 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nmqx\" (UniqueName: \"kubernetes.io/projected/fdd0a075-d5cf-4982-9623-c5432a308de6-kube-api-access-4nmqx\") pod \"maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr\" (UID: \"fdd0a075-d5cf-4982-9623-c5432a308de6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr" Oct 30 00:56:12.645492 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.645441 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:12.667423 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.667397 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr" Oct 30 00:56:12.870653 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.870574 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl" Oct 30 00:56:12.906320 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.906292 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq" Oct 30 00:56:12.938847 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.938815 5300 generic.go:356] "Generic (PLEG): container finished" podID="78eeb03d-9184-4a3d-af85-b8391731e828" containerID="596071fa4f369aa0913c47126c60790e71a4f450fc817aab4668ab7904f8522d" exitCode=0 Oct 30 00:56:12.938979 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.938948 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" event={"ID":"78eeb03d-9184-4a3d-af85-b8391731e828","Type":"ContainerDied","Data":"596071fa4f369aa0913c47126c60790e71a4f450fc817aab4668ab7904f8522d"} Oct 30 00:56:12.939388 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.939370 5300 scope.go:122] "RemoveContainer" containerID="596071fa4f369aa0913c47126c60790e71a4f450fc817aab4668ab7904f8522d" Oct 30 00:56:12.940911 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.940876 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:56:12.941005 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.940966 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:56:12.941057 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.941016 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:56:12.987773 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:12.987719 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr"] Oct 30 00:56:12.993044 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:56:12.993013 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd0a075_d5cf_4982_9623_c5432a308de6.slice/crio-f380c59eb34dd58c235f5696ea14e5457d6b9e1b7cb4e67bddf64796270587bc WatchSource:0}: Error finding container f380c59eb34dd58c235f5696ea14e5457d6b9e1b7cb4e67bddf64796270587bc: Status 404 returned error can't find the container with id f380c59eb34dd58c235f5696ea14e5457d6b9e1b7cb4e67bddf64796270587bc Oct 30 00:56:13.876624 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:13.876590 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw" Oct 30 00:56:13.954719 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:13.954681 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr" event={"ID":"fdd0a075-d5cf-4982-9623-c5432a308de6","Type":"ContainerStarted","Data":"764a2720e72cb36fb2ee7bc1d3074f71c8015bb427a8aec55b9847794d04ea3f"} Oct 30 00:56:13.954906 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:13.954732 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr" event={"ID":"fdd0a075-d5cf-4982-9623-c5432a308de6","Type":"ContainerStarted","Data":"f380c59eb34dd58c235f5696ea14e5457d6b9e1b7cb4e67bddf64796270587bc"} Oct 30 00:56:13.955456 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:13.955435 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr" Oct 30 00:56:13.957493 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:13.957470 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" event={"ID":"78eeb03d-9184-4a3d-af85-b8391731e828","Type":"ContainerStarted","Data":"2b4e8c4555bb286dae11f98457625a1e56a08fba51673d1c66aecdc08d31cbf8"} Oct 30 00:56:13.995459 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:13.995395 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr" podStartSLOduration=1.680899783 podStartE2EDuration="1.995375771s" totalImagesPullingTime="314.475988ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:13.99309919 +0000 UTC m=+774.308967517" watchObservedRunningTime="2026-10-30 00:56:13.995375771 +0000 UTC m=+774.311244097" Oct 30 00:56:14.180651 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:14.180544 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:14.452650 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:14.452566 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:14.556071 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:14.556033 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:56:14.556391 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:14.556363 5300 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 00:56:14.556462 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:14.556437 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" podUID="78eeb03d-9184-4a3d-af85-b8391731e828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 00:56:14.827499 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:14.827315 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:14.943733 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:14.943585 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:15.043820 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:15.041912 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:15.556387 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:15.556349 5300 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 00:56:15.556576 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:15.556463 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" podUID="78eeb03d-9184-4a3d-af85-b8391731e828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 00:56:15.891690 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:15.891602 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9" Oct 30 00:56:16.289545 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:16.289502 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:16.560053 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:16.559181 5300 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 00:56:16.560053 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:16.559248 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" podUID="78eeb03d-9184-4a3d-af85-b8391731e828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 00:56:17.066703 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:17.066667 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:17.560841 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:17.560144 5300 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 00:56:17.560841 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:17.560201 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" podUID="78eeb03d-9184-4a3d-af85-b8391731e828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 00:56:17.983176 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:17.983146 5300 generic.go:356] "Generic (PLEG): container finished" podID="ce797d22-c33f-4853-8c58-e014cd26bf5b" containerID="b6c96873414c39606d49caf52f7f34febe7ae307d4bc695f36d02bbf62dae394" exitCode=0 Oct 30 00:56:17.983359 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:17.983253 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" event={"ID":"ce797d22-c33f-4853-8c58-e014cd26bf5b","Type":"ContainerDied","Data":"b6c96873414c39606d49caf52f7f34febe7ae307d4bc695f36d02bbf62dae394"} Oct 30 00:56:17.983616 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:17.983601 5300 scope.go:122] "RemoveContainer" containerID="b6c96873414c39606d49caf52f7f34febe7ae307d4bc695f36d02bbf62dae394" Oct 30 00:56:17.984885 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:17.984853 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:56:17.984940 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:17.984923 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:56:17.984974 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:17.984956 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:56:18.444848 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:18.444579 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:18.562785 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:18.559240 5300 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 00:56:18.562785 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:18.559316 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" podUID="78eeb03d-9184-4a3d-af85-b8391731e828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 00:56:18.875785 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:18.863364 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:18.991568 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:18.991523 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" event={"ID":"ce797d22-c33f-4853-8c58-e014cd26bf5b","Type":"ContainerStarted","Data":"b1ca93273d8c925fca792624f4a23ffa5e1f986dc617b88f4446775f366a1446"} Oct 30 00:56:19.131837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:19.128117 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:56:19.132007 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:19.131902 5300 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 00:56:19.132007 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:19.131974 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" podUID="ce797d22-c33f-4853-8c58-e014cd26bf5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 00:56:19.369306 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:19.369266 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:19.477749 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:19.477720 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:19.596330 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:19.596290 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:19.600430 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:19.600394 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:56:19.600881 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:19.600863 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:56:19.602358 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:19.602339 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 00:56:19.974941 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:19.974346 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:20.075622 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:20.075577 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:20.138782 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:20.138280 5300 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 00:56:20.138782 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:20.138356 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" podUID="ce797d22-c33f-4853-8c58-e014cd26bf5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 00:56:20.178169 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:20.178140 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:20.727181 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:20.727146 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:20.979797 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:20.979304 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr" Oct 30 00:56:21.128239 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:21.128199 5300 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 00:56:21.128506 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:21.128474 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" podUID="ce797d22-c33f-4853-8c58-e014cd26bf5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 00:56:21.597984 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:56:21.597942 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:56:22.125785 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:22.122158 5300 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 00:56:22.125785 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:22.122225 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" podUID="ce797d22-c33f-4853-8c58-e014cd26bf5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 00:56:22.561784 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:22.561216 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:22.667787 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:22.661224 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:22.763785 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:22.763247 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:22.806449 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:22.806421 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:56:23.124171 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:23.124134 5300 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 00:56:23.124446 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:23.124415 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" podUID="ce797d22-c33f-4853-8c58-e014cd26bf5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 00:56:24.120200 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:24.120162 5300 patch_prober.go:29] interesting pod/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 00:56:24.120829 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:24.120792 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" podUID="ce797d22-c33f-4853-8c58-e014cd26bf5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 00:56:24.656439 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:24.656362 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:24.763622 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:24.763275 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:25.126548 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:25.126514 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:56:25.127538 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:25.127517 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 00:56:25.262398 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:25.262367 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:25.363472 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:25.363418 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:25.765027 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:25.764989 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:25.861971 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:25.861925 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:25.968571 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:25.968536 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:26.567240 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:26.567207 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:27.061810 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:27.061775 5300 generic.go:356] "Generic (PLEG): container finished" podID="6f99a456-9f20-4d79-950e-a79b016f5043" containerID="9405e2ce69ba988f41b1f3156dd2913008d77daf182dd0b6fa89adc2f23f58ba" exitCode=0 Oct 30 00:56:27.062001 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:27.061828 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" event={"ID":"6f99a456-9f20-4d79-950e-a79b016f5043","Type":"ContainerDied","Data":"9405e2ce69ba988f41b1f3156dd2913008d77daf182dd0b6fa89adc2f23f58ba"} Oct 30 00:56:27.062263 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:27.062248 5300 scope.go:122] "RemoveContainer" containerID="9405e2ce69ba988f41b1f3156dd2913008d77daf182dd0b6fa89adc2f23f58ba" Oct 30 00:56:27.063540 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:27.063515 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:56:27.063595 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:27.063581 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:56:27.063628 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:27.063615 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:56:28.068938 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:28.068902 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" event={"ID":"6f99a456-9f20-4d79-950e-a79b016f5043","Type":"ContainerStarted","Data":"04258f88fb36e4e8afb159aaf5d6994155ebcd5b46fa6b74231c3e9ab7fc0aca"} Oct 30 00:56:28.372848 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:28.372748 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:28.697013 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:28.696923 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:56:28.697159 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:28.697060 5300 patch_prober.go:29] interesting pod/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 00:56:28.697159 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:28.697128 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" podUID="6f99a456-9f20-4d79-950e-a79b016f5043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 00:56:28.987317 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:28.987280 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:29.652683 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:29.652626 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:29.698089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:29.698050 5300 patch_prober.go:29] interesting pod/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 00:56:29.698265 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:29.698119 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" podUID="6f99a456-9f20-4d79-950e-a79b016f5043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 00:56:29.757480 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:29.757443 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:30.244721 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:30.244687 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:30.344729 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:30.344702 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:30.444379 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:30.444319 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:30.697540 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:30.697461 5300 patch_prober.go:29] interesting pod/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 00:56:30.697540 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:30.697523 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" podUID="6f99a456-9f20-4d79-950e-a79b016f5043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 00:56:31.697640 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:31.697599 5300 patch_prober.go:29] interesting pod/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 00:56:31.697640 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:31.697655 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" podUID="6f99a456-9f20-4d79-950e-a79b016f5043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 00:56:32.306257 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:32.306226 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:32.697687 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:32.697603 5300 patch_prober.go:29] interesting pod/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 00:56:32.697687 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:32.697658 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" podUID="6f99a456-9f20-4d79-950e-a79b016f5043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 00:56:33.701181 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:33.701154 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:56:33.701555 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:33.701426 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:56:33.702054 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:33.702034 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 00:56:34.141084 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:34.141034 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:35.392978 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:35.392948 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:35.744529 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:35.744496 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:35.910575 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:35.910541 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct"] Oct 30 00:56:35.914401 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:35.914378 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 00:56:35.934486 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:35.934460 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-c4847c\"/\"kube-root-ca.crt\"" Oct 30 00:56:35.946419 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:35.946385 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-c4847c\"/\"openshift-service-ca.crt\"" Oct 30 00:56:35.946790 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:35.946755 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-c4847c\"/\"simulator-w6-c4847c-kserve-self-signed-certs\"" Oct 30 00:56:35.946916 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:35.946901 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-c4847c\"/\"default-dockercfg-w6qhr\"" Oct 30 00:56:35.949228 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:35.949203 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8h9n\" (UniqueName: \"kubernetes.io/projected/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-kube-api-access-z8h9n\") pod \"simulator-w6-c4847c-kserve-56bb64f4fd-6wrct\" (UID: \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\") " pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 00:56:35.949307 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:35.949249 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-model-cache\") pod \"simulator-w6-c4847c-kserve-56bb64f4fd-6wrct\" (UID: \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\") " pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 00:56:35.949364 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:35.949349 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-tls-certs\") pod \"simulator-w6-c4847c-kserve-56bb64f4fd-6wrct\" (UID: \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\") " pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 00:56:35.949401 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:35.949372 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-home\") pod \"simulator-w6-c4847c-kserve-56bb64f4fd-6wrct\" (UID: \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\") " pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 00:56:35.949454 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:35.949410 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-tmp-dir\") pod \"simulator-w6-c4847c-kserve-56bb64f4fd-6wrct\" (UID: \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\") " pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 00:56:35.949597 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:35.949571 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-dshm\") pod \"simulator-w6-c4847c-kserve-56bb64f4fd-6wrct\" (UID: \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\") " pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 00:56:35.950032 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:35.950012 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct"] Oct 30 00:56:35.983115 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:35.983084 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:36.050993 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:36.050911 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-model-cache\") pod \"simulator-w6-c4847c-kserve-56bb64f4fd-6wrct\" (UID: \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\") " pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 00:56:36.050993 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:36.050962 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-tls-certs\") pod \"simulator-w6-c4847c-kserve-56bb64f4fd-6wrct\" (UID: \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\") " pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 00:56:36.051229 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:36.050994 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-home\") pod \"simulator-w6-c4847c-kserve-56bb64f4fd-6wrct\" (UID: \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\") " pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 00:56:36.051229 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:36.051048 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-tmp-dir\") pod \"simulator-w6-c4847c-kserve-56bb64f4fd-6wrct\" (UID: \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\") " pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 00:56:36.051229 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:36.051094 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-dshm\") pod \"simulator-w6-c4847c-kserve-56bb64f4fd-6wrct\" (UID: \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\") " pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 00:56:36.051382 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:36.051230 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8h9n\" (UniqueName: \"kubernetes.io/projected/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-kube-api-access-z8h9n\") pod \"simulator-w6-c4847c-kserve-56bb64f4fd-6wrct\" (UID: \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\") " pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 00:56:36.051554 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:36.051491 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-tmp-dir\") pod \"simulator-w6-c4847c-kserve-56bb64f4fd-6wrct\" (UID: \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\") " pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 00:56:36.051554 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:36.051516 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-home\") pod \"simulator-w6-c4847c-kserve-56bb64f4fd-6wrct\" (UID: \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\") " pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 00:56:36.051732 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:36.051614 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-model-cache\") pod \"simulator-w6-c4847c-kserve-56bb64f4fd-6wrct\" (UID: \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\") " pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 00:56:36.053493 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:36.053470 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-dshm\") pod \"simulator-w6-c4847c-kserve-56bb64f4fd-6wrct\" (UID: \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\") " pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 00:56:36.053661 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:36.053644 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-tls-certs\") pod \"simulator-w6-c4847c-kserve-56bb64f4fd-6wrct\" (UID: \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\") " pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 00:56:36.078360 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:36.078330 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8h9n\" (UniqueName: \"kubernetes.io/projected/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-kube-api-access-z8h9n\") pod \"simulator-w6-c4847c-kserve-56bb64f4fd-6wrct\" (UID: \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\") " pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 00:56:36.097079 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:36.097057 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:36.197412 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:36.197377 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:36.228982 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:36.228942 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 00:56:36.557829 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:36.557797 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct"] Oct 30 00:56:36.561669 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:56:36.561635 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode51b311d_e368_4ecc_80a7_d1cb4a4ed8b7.slice/crio-226026643f758e2df9dbc3de29ff2da11584339cf435cb4e320ec7be5798f8f2 WatchSource:0}: Error finding container 226026643f758e2df9dbc3de29ff2da11584339cf435cb4e320ec7be5798f8f2: Status 404 returned error can't find the container with id 226026643f758e2df9dbc3de29ff2da11584339cf435cb4e320ec7be5798f8f2 Oct 30 00:56:37.126323 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:37.126286 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" event={"ID":"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7","Type":"ContainerStarted","Data":"f2f96c802581b72e7deb54df346e854e292c345b6d55d991851c1552a4596760"} Oct 30 00:56:37.126514 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:37.126333 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" event={"ID":"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7","Type":"ContainerStarted","Data":"226026643f758e2df9dbc3de29ff2da11584339cf435cb4e320ec7be5798f8f2"} Oct 30 00:56:37.166857 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:37.166802 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podStartSLOduration=2.166787361 podStartE2EDuration="2.166787361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:37.165102521 +0000 UTC m=+797.480970847" watchObservedRunningTime="2026-10-30 00:56:37.166787361 +0000 UTC m=+797.482655686" Oct 30 00:56:37.229294 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:37.229258 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 00:56:37.485707 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:37.485674 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:37.662635 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:37.662597 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:39.435464 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:39.435418 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:41.068684 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:41.068646 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:42.841787 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:42.841381 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:43.444434 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:43.444401 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:44.639648 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:44.639603 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:46.206697 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:46.206660 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:46.229836 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:46.229800 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 00:56:46.242836 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:46.242812 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 00:56:47.198649 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:47.198622 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 00:56:47.950259 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:47.950217 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:49.643216 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:49.643182 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:51.246437 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:51.246405 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:51.329715 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:51.329669 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:51.592729 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:56:51.592649 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:56:53.038887 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:53.038843 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:54.650795 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:54.650723 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:55.958973 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:55.958935 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg"] Oct 30 00:56:55.964927 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:55.964901 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 00:56:56.001172 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:56.001133 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg"] Oct 30 00:56:56.001495 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:56.001447 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c104b8\"/\"kube-root-ca.crt\"" Oct 30 00:56:56.001721 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:56.001705 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c104b8\"/\"default-dockercfg-2kgzn\"" Oct 30 00:56:56.012795 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:56.012731 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-tmp-dir\") pod \"premium-w1-c104b8-kserve-6d469c76d7-pvgrg\" (UID: \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\") " pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 00:56:56.013334 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:56.013308 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-dshm\") pod \"premium-w1-c104b8-kserve-6d469c76d7-pvgrg\" (UID: \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\") " pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 00:56:56.013524 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:56.013506 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgj5b\" (UniqueName: \"kubernetes.io/projected/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-kube-api-access-hgj5b\") pod \"premium-w1-c104b8-kserve-6d469c76d7-pvgrg\" (UID: \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\") " pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 00:56:56.013924 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:56.013901 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-tls-certs\") pod \"premium-w1-c104b8-kserve-6d469c76d7-pvgrg\" (UID: \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\") " pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 00:56:56.014088 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:56.014069 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-home\") pod \"premium-w1-c104b8-kserve-6d469c76d7-pvgrg\" (UID: \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\") " pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 00:56:56.014167 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:56.013246 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c104b8\"/\"premium-w1-c104b8-kserve-self-signed-certs\"" Oct 30 00:56:56.014167 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:56.014121 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-model-cache\") pod \"premium-w1-c104b8-kserve-6d469c76d7-pvgrg\" (UID: \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\") " pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 00:56:56.014167 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:56.013152 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c104b8\"/\"openshift-service-ca.crt\"" Oct 30 00:56:56.105143 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:56.105111 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:56.115480 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:56.115450 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-tls-certs\") pod \"premium-w1-c104b8-kserve-6d469c76d7-pvgrg\" (UID: \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\") " pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 00:56:56.115600 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:56.115502 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-home\") pod \"premium-w1-c104b8-kserve-6d469c76d7-pvgrg\" (UID: \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\") " pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 00:56:56.115600 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:56.115538 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-model-cache\") pod \"premium-w1-c104b8-kserve-6d469c76d7-pvgrg\" (UID: \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\") " pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 00:56:56.115713 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:56.115622 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-tmp-dir\") pod \"premium-w1-c104b8-kserve-6d469c76d7-pvgrg\" (UID: \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\") " pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 00:56:56.115785 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:56.115755 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-dshm\") pod \"premium-w1-c104b8-kserve-6d469c76d7-pvgrg\" (UID: \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\") " pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 00:56:56.115861 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:56.115839 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgj5b\" (UniqueName: \"kubernetes.io/projected/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-kube-api-access-hgj5b\") pod \"premium-w1-c104b8-kserve-6d469c76d7-pvgrg\" (UID: \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\") " pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 00:56:56.116038 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:56.116017 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-model-cache\") pod \"premium-w1-c104b8-kserve-6d469c76d7-pvgrg\" (UID: \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\") " pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 00:56:56.116249 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:56.116230 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-home\") pod \"premium-w1-c104b8-kserve-6d469c76d7-pvgrg\" (UID: \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\") " pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 00:56:56.116326 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:56.116296 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-tmp-dir\") pod \"premium-w1-c104b8-kserve-6d469c76d7-pvgrg\" (UID: \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\") " pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 00:56:56.118261 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:56.118240 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-dshm\") pod \"premium-w1-c104b8-kserve-6d469c76d7-pvgrg\" (UID: \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\") " pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 00:56:56.118353 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:56.118342 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-tls-certs\") pod \"premium-w1-c104b8-kserve-6d469c76d7-pvgrg\" (UID: \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\") " pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 00:56:56.136310 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:56.136286 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgj5b\" (UniqueName: \"kubernetes.io/projected/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-kube-api-access-hgj5b\") pod \"premium-w1-c104b8-kserve-6d469c76d7-pvgrg\" (UID: \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\") " pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 00:56:56.330999 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:56.330963 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 00:56:56.454111 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:56.454077 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:56.627364 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:56.627290 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg"] Oct 30 00:56:56.630504 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:56:56.630470 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4df2fe3_8896_48bf_afdd_bdf13ca5d4a9.slice/crio-f0816c29b647986ef4c9f38c0402bc2bb9d3ec5c7e5b1553f068922bc290b8ab WatchSource:0}: Error finding container f0816c29b647986ef4c9f38c0402bc2bb9d3ec5c7e5b1553f068922bc290b8ab: Status 404 returned error can't find the container with id f0816c29b647986ef4c9f38c0402bc2bb9d3ec5c7e5b1553f068922bc290b8ab Oct 30 00:56:57.250429 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:57.250386 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" event={"ID":"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9","Type":"ContainerStarted","Data":"08a8a17422c6da0dda5722c19a464b5c5e4db161e9638a70759c8289e29709e0"} Oct 30 00:56:57.250852 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:57.250438 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" event={"ID":"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9","Type":"ContainerStarted","Data":"f0816c29b647986ef4c9f38c0402bc2bb9d3ec5c7e5b1553f068922bc290b8ab"} Oct 30 00:56:57.288736 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:57.288657 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" podStartSLOduration=2.288586403 podStartE2EDuration="2.288586403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:57.287741062 +0000 UTC m=+817.603609386" watchObservedRunningTime="2026-10-30 00:56:57.288586403 +0000 UTC m=+817.604454728" Oct 30 00:56:57.331103 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:57.331070 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 00:56:58.154511 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:58.154454 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:59.179678 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:59.179646 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:56:59.766390 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:56:59.766336 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:01.305246 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:01.305165 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:03.138772 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:03.138734 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:04.879473 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:04.879404 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:06.331265 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:06.331221 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 00:57:06.346019 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:06.345992 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 00:57:06.604512 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:06.604412 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:07.314833 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.314804 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 00:57:07.525637 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.525605 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q"] Oct 30 00:57:07.529616 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.529598 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 00:57:07.547550 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.547522 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q"] Oct 30 00:57:07.547959 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.547935 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-dca38d\"/\"kube-root-ca.crt\"" Oct 30 00:57:07.556325 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.556295 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-dca38d\"/\"default-dockercfg-c5kjw\"" Oct 30 00:57:07.556325 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.556313 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-dca38d\"/\"premium-w5-dca38d-kserve-self-signed-certs\"" Oct 30 00:57:07.556325 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.556304 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-dca38d\"/\"openshift-service-ca.crt\"" Oct 30 00:57:07.573964 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.573907 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:07.639742 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.639708 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-dshm\") pod \"premium-w5-dca38d-kserve-6c579c47c4-mqn6q\" (UID: \"4d4a7520-7709-4747-b6b1-dda7c86290c1\") " pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 00:57:07.639742 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.639745 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-home\") pod \"premium-w5-dca38d-kserve-6c579c47c4-mqn6q\" (UID: \"4d4a7520-7709-4747-b6b1-dda7c86290c1\") " pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 00:57:07.639968 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.639874 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4a7520-7709-4747-b6b1-dda7c86290c1-tls-certs\") pod \"premium-w5-dca38d-kserve-6c579c47c4-mqn6q\" (UID: \"4d4a7520-7709-4747-b6b1-dda7c86290c1\") " pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 00:57:07.639968 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.639901 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-model-cache\") pod \"premium-w5-dca38d-kserve-6c579c47c4-mqn6q\" (UID: \"4d4a7520-7709-4747-b6b1-dda7c86290c1\") " pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 00:57:07.639968 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.639919 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g28nl\" (UniqueName: \"kubernetes.io/projected/4d4a7520-7709-4747-b6b1-dda7c86290c1-kube-api-access-g28nl\") pod \"premium-w5-dca38d-kserve-6c579c47c4-mqn6q\" (UID: \"4d4a7520-7709-4747-b6b1-dda7c86290c1\") " pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 00:57:07.640064 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.640042 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-tmp-dir\") pod \"premium-w5-dca38d-kserve-6c579c47c4-mqn6q\" (UID: \"4d4a7520-7709-4747-b6b1-dda7c86290c1\") " pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 00:57:07.741512 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.741449 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-dshm\") pod \"premium-w5-dca38d-kserve-6c579c47c4-mqn6q\" (UID: \"4d4a7520-7709-4747-b6b1-dda7c86290c1\") " pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 00:57:07.741512 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.741510 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-home\") pod \"premium-w5-dca38d-kserve-6c579c47c4-mqn6q\" (UID: \"4d4a7520-7709-4747-b6b1-dda7c86290c1\") " pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 00:57:07.741832 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.741574 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4a7520-7709-4747-b6b1-dda7c86290c1-tls-certs\") pod \"premium-w5-dca38d-kserve-6c579c47c4-mqn6q\" (UID: \"4d4a7520-7709-4747-b6b1-dda7c86290c1\") " pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 00:57:07.741832 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.741608 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-model-cache\") pod \"premium-w5-dca38d-kserve-6c579c47c4-mqn6q\" (UID: \"4d4a7520-7709-4747-b6b1-dda7c86290c1\") " pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 00:57:07.741832 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.741636 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g28nl\" (UniqueName: \"kubernetes.io/projected/4d4a7520-7709-4747-b6b1-dda7c86290c1-kube-api-access-g28nl\") pod \"premium-w5-dca38d-kserve-6c579c47c4-mqn6q\" (UID: \"4d4a7520-7709-4747-b6b1-dda7c86290c1\") " pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 00:57:07.741832 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.741727 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-tmp-dir\") pod \"premium-w5-dca38d-kserve-6c579c47c4-mqn6q\" (UID: \"4d4a7520-7709-4747-b6b1-dda7c86290c1\") " pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 00:57:07.742385 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.742267 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-tmp-dir\") pod \"premium-w5-dca38d-kserve-6c579c47c4-mqn6q\" (UID: \"4d4a7520-7709-4747-b6b1-dda7c86290c1\") " pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 00:57:07.742541 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.742517 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-model-cache\") pod \"premium-w5-dca38d-kserve-6c579c47c4-mqn6q\" (UID: \"4d4a7520-7709-4747-b6b1-dda7c86290c1\") " pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 00:57:07.743405 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.743380 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-home\") pod \"premium-w5-dca38d-kserve-6c579c47c4-mqn6q\" (UID: \"4d4a7520-7709-4747-b6b1-dda7c86290c1\") " pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 00:57:07.745593 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.745296 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-dshm\") pod \"premium-w5-dca38d-kserve-6c579c47c4-mqn6q\" (UID: \"4d4a7520-7709-4747-b6b1-dda7c86290c1\") " pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 00:57:07.747628 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.747581 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4a7520-7709-4747-b6b1-dda7c86290c1-tls-certs\") pod \"premium-w5-dca38d-kserve-6c579c47c4-mqn6q\" (UID: \"4d4a7520-7709-4747-b6b1-dda7c86290c1\") " pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 00:57:07.762339 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.762313 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g28nl\" (UniqueName: \"kubernetes.io/projected/4d4a7520-7709-4747-b6b1-dda7c86290c1-kube-api-access-g28nl\") pod \"premium-w5-dca38d-kserve-6c579c47c4-mqn6q\" (UID: \"4d4a7520-7709-4747-b6b1-dda7c86290c1\") " pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 00:57:07.843972 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:07.843887 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 00:57:08.139147 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:08.139070 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q"] Oct 30 00:57:08.142208 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:57:08.142176 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d4a7520_7709_4747_b6b1_dda7c86290c1.slice/crio-45b234972420fd72874f4dd6a99db68269aa16ddfbebe72f330f56b7dae72a83 WatchSource:0}: Error finding container 45b234972420fd72874f4dd6a99db68269aa16ddfbebe72f330f56b7dae72a83: Status 404 returned error can't find the container with id 45b234972420fd72874f4dd6a99db68269aa16ddfbebe72f330f56b7dae72a83 Oct 30 00:57:08.249538 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:08.249497 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:08.312516 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:08.312467 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" event={"ID":"4d4a7520-7709-4747-b6b1-dda7c86290c1","Type":"ContainerStarted","Data":"ba2aa076b6b05abc96d46bacb29fc7ea1d4d0c199e211046a9de96b50c20583d"} Oct 30 00:57:08.312728 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:08.312529 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" event={"ID":"4d4a7520-7709-4747-b6b1-dda7c86290c1","Type":"ContainerStarted","Data":"45b234972420fd72874f4dd6a99db68269aa16ddfbebe72f330f56b7dae72a83"} Oct 30 00:57:08.351041 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:08.350993 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" podStartSLOduration=1.350978747 podStartE2EDuration="1.350978747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:08.350194064 +0000 UTC m=+828.666062390" watchObservedRunningTime="2026-10-30 00:57:08.350978747 +0000 UTC m=+828.666847071" Oct 30 00:57:08.844974 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:08.844936 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 00:57:09.904571 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:09.904534 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:10.010886 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:10.010850 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:11.646721 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:11.646683 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:13.214866 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:13.214832 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:14.760629 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:14.760595 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:16.103175 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:16.103137 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:16.767837 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:16.767801 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:17.844850 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:17.844818 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 00:57:17.857988 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:17.857953 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 00:57:18.429173 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.429146 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 00:57:18.448118 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.448064 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:18.534161 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.534127 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms"] Oct 30 00:57:18.539586 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.539561 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 00:57:18.557626 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.557598 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms"] Oct 30 00:57:18.567130 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.567106 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-82b647\"/\"openshift-service-ca.crt\"" Oct 30 00:57:18.567130 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.567106 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-82b647\"/\"default-dockercfg-g24xb\"" Oct 30 00:57:18.567130 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.567110 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-82b647\"/\"premium-w3-82b647-kserve-self-signed-certs\"" Oct 30 00:57:18.569368 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.569348 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-82b647\"/\"kube-root-ca.crt\"" Oct 30 00:57:18.583005 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.582973 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:18.680346 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.680260 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da1ea263-175e-4b07-b74d-a30b16d9ed8c-tls-certs\") pod \"premium-w3-82b647-kserve-c9979684b-j6hms\" (UID: \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\") " pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 00:57:18.680346 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.680329 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-tmp-dir\") pod \"premium-w3-82b647-kserve-c9979684b-j6hms\" (UID: \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\") " pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 00:57:18.680549 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.680399 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n9xc\" (UniqueName: \"kubernetes.io/projected/da1ea263-175e-4b07-b74d-a30b16d9ed8c-kube-api-access-4n9xc\") pod \"premium-w3-82b647-kserve-c9979684b-j6hms\" (UID: \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\") " pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 00:57:18.680549 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.680432 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-home\") pod \"premium-w3-82b647-kserve-c9979684b-j6hms\" (UID: \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\") " pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 00:57:18.680549 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.680459 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-dshm\") pod \"premium-w3-82b647-kserve-c9979684b-j6hms\" (UID: \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\") " pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 00:57:18.680668 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.680576 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-model-cache\") pod \"premium-w3-82b647-kserve-c9979684b-j6hms\" (UID: \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\") " pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 00:57:18.782201 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.782163 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-home\") pod \"premium-w3-82b647-kserve-c9979684b-j6hms\" (UID: \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\") " pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 00:57:18.782389 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.782215 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-dshm\") pod \"premium-w3-82b647-kserve-c9979684b-j6hms\" (UID: \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\") " pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 00:57:18.782389 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.782265 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-model-cache\") pod \"premium-w3-82b647-kserve-c9979684b-j6hms\" (UID: \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\") " pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 00:57:18.782389 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.782332 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da1ea263-175e-4b07-b74d-a30b16d9ed8c-tls-certs\") pod \"premium-w3-82b647-kserve-c9979684b-j6hms\" (UID: \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\") " pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 00:57:18.782561 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.782391 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-tmp-dir\") pod \"premium-w3-82b647-kserve-c9979684b-j6hms\" (UID: \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\") " pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 00:57:18.782561 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.782442 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n9xc\" (UniqueName: \"kubernetes.io/projected/da1ea263-175e-4b07-b74d-a30b16d9ed8c-kube-api-access-4n9xc\") pod \"premium-w3-82b647-kserve-c9979684b-j6hms\" (UID: \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\") " pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 00:57:18.782710 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.782683 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-home\") pod \"premium-w3-82b647-kserve-c9979684b-j6hms\" (UID: \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\") " pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 00:57:18.782805 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.782739 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-model-cache\") pod \"premium-w3-82b647-kserve-c9979684b-j6hms\" (UID: \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\") " pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 00:57:18.782805 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.782794 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-tmp-dir\") pod \"premium-w3-82b647-kserve-c9979684b-j6hms\" (UID: \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\") " pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 00:57:18.784506 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.784481 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-dshm\") pod \"premium-w3-82b647-kserve-c9979684b-j6hms\" (UID: \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\") " pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 00:57:18.784770 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.784740 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da1ea263-175e-4b07-b74d-a30b16d9ed8c-tls-certs\") pod \"premium-w3-82b647-kserve-c9979684b-j6hms\" (UID: \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\") " pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 00:57:18.801405 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.801376 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n9xc\" (UniqueName: \"kubernetes.io/projected/da1ea263-175e-4b07-b74d-a30b16d9ed8c-kube-api-access-4n9xc\") pod \"premium-w3-82b647-kserve-c9979684b-j6hms\" (UID: \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\") " pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 00:57:18.854467 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:18.854435 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 00:57:19.154483 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:19.154454 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms"] Oct 30 00:57:19.157073 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:57:19.157039 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda1ea263_175e_4b07_b74d_a30b16d9ed8c.slice/crio-93f7a209a6eac153b0749608017bab4ccbd94abc1d46cf89e3370556f9ede1e5 WatchSource:0}: Error finding container 93f7a209a6eac153b0749608017bab4ccbd94abc1d46cf89e3370556f9ede1e5: Status 404 returned error can't find the container with id 93f7a209a6eac153b0749608017bab4ccbd94abc1d46cf89e3370556f9ede1e5 Oct 30 00:57:19.424802 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:19.424699 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" event={"ID":"da1ea263-175e-4b07-b74d-a30b16d9ed8c","Type":"ContainerStarted","Data":"9f42a87c220017a47e98887cd5d046a55c03878453153acf49414a9ea13a7b04"} Oct 30 00:57:19.424802 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:19.424753 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" event={"ID":"da1ea263-175e-4b07-b74d-a30b16d9ed8c","Type":"ContainerStarted","Data":"93f7a209a6eac153b0749608017bab4ccbd94abc1d46cf89e3370556f9ede1e5"} Oct 30 00:57:19.467205 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:19.467112 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podStartSLOduration=1.467094085 podStartE2EDuration="1.467094085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:19.465573397 +0000 UTC m=+839.781441722" watchObservedRunningTime="2026-10-30 00:57:19.467094085 +0000 UTC m=+839.782962411" Oct 30 00:57:19.854925 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:19.854894 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 00:57:20.133789 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:20.133683 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:21.594636 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:57:21.594594 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:57:21.917950 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:21.917869 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:23.559072 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:23.559033 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:25.160575 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:25.160545 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:25.239712 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:25.239678 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:27.073225 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:27.073190 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:28.702340 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:28.702293 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:28.854819 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:28.854781 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 00:57:28.868005 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:28.867979 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 00:57:29.493786 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:29.493736 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 00:57:29.794292 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:29.794200 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:30.487572 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:30.487529 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:32.080805 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:32.080755 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:32.197398 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:32.197358 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:33.866833 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:33.866802 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:35.590542 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:35.590491 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:37.497128 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:37.497095 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:39.340314 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:39.340284 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:40.996135 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:40.996104 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:42.547930 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:42.547894 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:44.142186 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:44.142153 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:45.771115 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:45.771078 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:47.376674 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:47.376642 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:49.065663 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:49.065626 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:50.860019 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:50.859920 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:51.013445 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.013410 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq"] Oct 30 00:57:51.018560 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.018538 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.036469 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.036446 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-c4847c-openshift-default-dockercfg-4nf8k\"" Oct 30 00:57:51.051255 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.051225 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq"] Oct 30 00:57:51.164428 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.164330 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.164428 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.164377 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.164428 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.164402 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.164684 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.164541 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.164684 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.164587 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.164684 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.164649 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6m2p\" (UniqueName: \"kubernetes.io/projected/e7d2e040-342a-4ea4-a23d-c36722f2d64e-kube-api-access-h6m2p\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.164883 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.164799 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.164920 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.164901 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.164968 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.164935 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.265665 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.265629 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.265894 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.265676 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.265894 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.265727 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.265894 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.265751 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.265894 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.265790 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.265894 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.265857 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.265894 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.265883 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.266244 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.265987 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6m2p\" (UniqueName: \"kubernetes.io/projected/e7d2e040-342a-4ea4-a23d-c36722f2d64e-kube-api-access-h6m2p\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.266244 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.266037 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.266244 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.266140 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.266480 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.266454 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.266552 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.266460 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.266613 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.266545 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.266777 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.266684 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.267288 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.267261 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:51.269281 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.269074 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.269281 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.269239 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.290447 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.290416 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.292922 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.292893 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6m2p\" (UniqueName: \"kubernetes.io/projected/e7d2e040-342a-4ea4-a23d-c36722f2d64e-kube-api-access-h6m2p\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.331595 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.331554 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:51.596559 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:57:51.596515 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:57:51.660575 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.660533 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq"] Oct 30 00:57:51.683723 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.682393 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:57:51.683723 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.682487 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:57:51.683723 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:51.682574 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:57:52.609172 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:52.609132 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" event={"ID":"e7d2e040-342a-4ea4-a23d-c36722f2d64e","Type":"ContainerStarted","Data":"7179cad9c68a74a0dd3340ec7249dd966f39f34e243f2b71c3fe29e7aebc9446"} Oct 30 00:57:52.609551 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:52.609194 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" event={"ID":"e7d2e040-342a-4ea4-a23d-c36722f2d64e","Type":"ContainerStarted","Data":"a0fa3bf0588a27cdb98be01cc394038d8009ed74d55077f522f6236e88cdec66"} Oct 30 00:57:52.616956 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:52.616930 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:52.692795 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:52.692712 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" podStartSLOduration=2.692692589 podStartE2EDuration="2.692692589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:52.676950354 +0000 UTC m=+872.992818698" watchObservedRunningTime="2026-10-30 00:57:52.692692589 +0000 UTC m=+873.008560915" Oct 30 00:57:53.339785 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:53.338137 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:53.343388 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:53.343355 5300 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 00:57:53.343568 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:53.343433 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" podUID="e7d2e040-342a-4ea4-a23d-c36722f2d64e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 00:57:54.343788 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:54.334215 5300 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 00:57:54.343788 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:54.334281 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" podUID="e7d2e040-342a-4ea4-a23d-c36722f2d64e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 00:57:54.482375 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:54.482291 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:55.339786 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:55.339202 5300 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 00:57:55.339786 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:55.339282 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" podUID="e7d2e040-342a-4ea4-a23d-c36722f2d64e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 00:57:56.288522 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:56.288486 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:56.336300 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:56.336259 5300 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 00:57:56.336574 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:56.336545 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" podUID="e7d2e040-342a-4ea4-a23d-c36722f2d64e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 00:57:57.332787 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:57.332034 5300 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 00:57:57.332787 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:57.332122 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" podUID="e7d2e040-342a-4ea4-a23d-c36722f2d64e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 00:57:58.145503 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:58.145470 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:57:58.556074 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:58.554496 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:58.556074 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:58.555082 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:58.557130 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:58.557105 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 00:57:59.918903 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:57:59.918848 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:01.599053 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:01.599018 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:03.275487 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:03.275441 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:04.920087 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:04.920044 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:05.928211 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:05.928172 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8"] Oct 30 00:58:05.932175 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:05.932150 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:05.948289 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:05.948264 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d827b7-openshift-default-dockercfg-95vfg\"" Oct 30 00:58:05.963826 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:05.963791 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8"] Oct 30 00:58:05.989794 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:05.989736 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:05.990000 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:05.989855 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db62d92e-f4b6-480b-9545-569baf9670a8-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:05.990000 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:05.989886 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:05.990000 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:05.989920 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:05.990194 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:05.990041 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:05.990194 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:05.990095 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km6w9\" (UniqueName: \"kubernetes.io/projected/db62d92e-f4b6-480b-9545-569baf9670a8-kube-api-access-km6w9\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:05.990194 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:05.990185 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db62d92e-f4b6-480b-9545-569baf9670a8-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:05.990353 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:05.990240 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db62d92e-f4b6-480b-9545-569baf9670a8-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:05.990353 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:05.990273 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:06.091831 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:06.091751 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:06.091831 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:06.091837 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:06.092076 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:06.091867 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:06.092076 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:06.091903 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km6w9\" (UniqueName: \"kubernetes.io/projected/db62d92e-f4b6-480b-9545-569baf9670a8-kube-api-access-km6w9\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:06.092184 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:06.092152 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db62d92e-f4b6-480b-9545-569baf9670a8-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:06.092258 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:06.092241 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db62d92e-f4b6-480b-9545-569baf9670a8-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:06.092380 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:06.092354 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:06.092465 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:06.092429 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:06.092465 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:06.092442 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:06.092575 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:06.092510 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:06.092631 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:06.092586 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db62d92e-f4b6-480b-9545-569baf9670a8-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:06.092741 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:06.092714 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:06.093094 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:06.093061 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db62d92e-f4b6-480b-9545-569baf9670a8-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:06.093267 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:06.093238 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:06.094657 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:06.094631 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:06.094991 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:06.094970 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db62d92e-f4b6-480b-9545-569baf9670a8-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:06.141818 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:06.129878 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km6w9\" (UniqueName: \"kubernetes.io/projected/db62d92e-f4b6-480b-9545-569baf9670a8-kube-api-access-km6w9\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:06.141818 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:06.136067 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db62d92e-f4b6-480b-9545-569baf9670a8-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:06.245836 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:06.245792 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:06.562506 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:06.561277 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8"] Oct 30 00:58:06.564580 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:58:06.564544 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb62d92e_f4b6_480b_9545_569baf9670a8.slice/crio-f95f96b2a6f30bbf671492b34139903272ecc8a206c382a3572d73069614fd99 WatchSource:0}: Error finding container f95f96b2a6f30bbf671492b34139903272ecc8a206c382a3572d73069614fd99: Status 404 returned error can't find the container with id f95f96b2a6f30bbf671492b34139903272ecc8a206c382a3572d73069614fd99 Oct 30 00:58:06.567354 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:06.567320 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:58:06.567499 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:06.567396 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:58:06.567499 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:06.567427 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:58:06.715414 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:06.715369 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:06.728075 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:06.728037 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" event={"ID":"db62d92e-f4b6-480b-9545-569baf9670a8","Type":"ContainerStarted","Data":"e3f0420d312b02f08636fe9caf4ea8fb2200987eba800ca086ff703ebc5a6fa3"} Oct 30 00:58:06.728276 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:06.728099 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" event={"ID":"db62d92e-f4b6-480b-9545-569baf9670a8","Type":"ContainerStarted","Data":"f95f96b2a6f30bbf671492b34139903272ecc8a206c382a3572d73069614fd99"} Oct 30 00:58:06.771941 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:06.771886 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" podStartSLOduration=1.771870552 podStartE2EDuration="1.771870552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:06.769815679 +0000 UTC m=+887.085684030" watchObservedRunningTime="2026-10-30 00:58:06.771870552 +0000 UTC m=+887.087738877" Oct 30 00:58:07.246781 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:07.246173 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:07.255781 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:07.248628 5300 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 00:58:07.255781 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:07.248687 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" podUID="db62d92e-f4b6-480b-9545-569baf9670a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 00:58:08.246786 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:08.246239 5300 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 00:58:08.246786 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:08.246314 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" podUID="db62d92e-f4b6-480b-9545-569baf9670a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 00:58:08.619839 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:08.618343 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:09.253784 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:09.250218 5300 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 00:58:09.253784 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:09.250300 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" podUID="db62d92e-f4b6-480b-9545-569baf9670a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 00:58:10.262736 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:10.262248 5300 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 00:58:10.262736 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:10.262315 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" podUID="db62d92e-f4b6-480b-9545-569baf9670a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 00:58:10.767282 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:10.767217 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:11.246670 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:11.246632 5300 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 00:58:11.246881 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:11.246688 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" podUID="db62d92e-f4b6-480b-9545-569baf9670a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 00:58:12.246389 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:12.246346 5300 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 00:58:12.246828 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:12.246404 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" podUID="db62d92e-f4b6-480b-9545-569baf9670a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 00:58:12.505579 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:12.505490 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:13.246574 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:13.246536 5300 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 00:58:13.247056 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:13.246594 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" podUID="db62d92e-f4b6-480b-9545-569baf9670a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 00:58:14.250674 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:14.250644 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:14.251119 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:14.251032 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:14.251750 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:14.251728 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 00:58:14.275158 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:14.275129 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:14.379200 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:14.379060 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:16.019011 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:16.018963 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:17.622906 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:17.622851 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:19.345699 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:19.345664 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:20.478698 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.478527 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp"] Oct 30 00:58:20.512796 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.509733 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.514973 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.514951 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp"] Oct 30 00:58:20.525420 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.525397 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-dca38d-openshift-default-dockercfg-b9zlp\"" Oct 30 00:58:20.585468 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.585418 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.585794 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.585751 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.585976 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.585953 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppdnf\" (UniqueName: \"kubernetes.io/projected/12e75f96-d218-400a-8b56-0cea344f3e41-kube-api-access-ppdnf\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.586089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.586075 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e75f96-d218-400a-8b56-0cea344f3e41-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.586247 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.586232 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.586351 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.586338 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12e75f96-d218-400a-8b56-0cea344f3e41-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.586450 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.586438 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.586571 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.586559 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12e75f96-d218-400a-8b56-0cea344f3e41-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.586674 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.586661 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.688278 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.688241 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.688468 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.688301 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppdnf\" (UniqueName: \"kubernetes.io/projected/12e75f96-d218-400a-8b56-0cea344f3e41-kube-api-access-ppdnf\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.688468 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.688323 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e75f96-d218-400a-8b56-0cea344f3e41-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.688468 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.688370 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.688468 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.688391 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12e75f96-d218-400a-8b56-0cea344f3e41-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.688468 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.688411 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.688468 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.688444 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12e75f96-d218-400a-8b56-0cea344f3e41-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.688468 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.688467 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.688802 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.688535 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.688802 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.688721 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.688957 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.688882 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.689173 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.689152 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.689458 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.689430 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.689637 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.689615 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12e75f96-d218-400a-8b56-0cea344f3e41-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.691525 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.691501 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12e75f96-d218-400a-8b56-0cea344f3e41-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.691625 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.691509 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.708442 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.708412 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e75f96-d218-400a-8b56-0cea344f3e41-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.708874 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.708847 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppdnf\" (UniqueName: \"kubernetes.io/projected/12e75f96-d218-400a-8b56-0cea344f3e41-kube-api-access-ppdnf\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.857173 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.857085 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:20.995826 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:20.995785 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:21.174171 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:21.174061 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp"] Oct 30 00:58:21.178614 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:58:21.178569 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e75f96_d218_400a_8b56_0cea344f3e41.slice/crio-6862b53370e72768353a5ac9d9b326420ee8ae0761bef77ad426c3b6b1091740 WatchSource:0}: Error finding container 6862b53370e72768353a5ac9d9b326420ee8ae0761bef77ad426c3b6b1091740: Status 404 returned error can't find the container with id 6862b53370e72768353a5ac9d9b326420ee8ae0761bef77ad426c3b6b1091740 Oct 30 00:58:21.181494 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:21.181447 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:58:21.181590 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:21.181551 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:58:21.181628 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:21.181594 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:58:21.594947 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:58:21.594906 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:58:21.890021 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:21.889913 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" event={"ID":"12e75f96-d218-400a-8b56-0cea344f3e41","Type":"ContainerStarted","Data":"1a0e1feca9b621ad561013ee0cccc5b6caa47cdb9b0fdd18af45b6e5c7b863a1"} Oct 30 00:58:21.890321 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:21.890295 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" event={"ID":"12e75f96-d218-400a-8b56-0cea344f3e41","Type":"ContainerStarted","Data":"6862b53370e72768353a5ac9d9b326420ee8ae0761bef77ad426c3b6b1091740"} Oct 30 00:58:22.864781 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:22.864732 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:22.875781 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:22.867024 5300 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 00:58:22.875781 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:22.867085 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" podUID="12e75f96-d218-400a-8b56-0cea344f3e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 00:58:23.317785 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:23.314862 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:23.417333 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:23.417272 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" podUID="1166fd98-968e-4fda-a0b9-0cfea1ed9e03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 00:58:23.422158 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:23.421824 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:23.607971 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:23.607855 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 00:58:23.878847 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:23.865867 5300 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 00:58:23.878847 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:23.865933 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" podUID="12e75f96-d218-400a-8b56-0cea344f3e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 00:58:24.188964 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:24.188849 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:24.324910 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:24.324854 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 00:58:24.325827 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:24.325803 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 00:58:24.374495 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:24.374424 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" podStartSLOduration=4.374402132 podStartE2EDuration="4.374402132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:21.983148443 +0000 UTC m=+902.299016768" watchObservedRunningTime="2026-10-30 00:58:24.374402132 +0000 UTC m=+904.690270460" Oct 30 00:58:24.432905 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:24.432857 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" podUID="1166fd98-968e-4fda-a0b9-0cfea1ed9e03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:24.444836 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:24.433727 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 00:58:24.602051 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:24.451074 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:24.613790 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:24.611812 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 00:58:24.622789 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:24.620134 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 00:58:24.637892 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:24.637867 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 00:58:24.911107 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:24.911069 5300 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 00:58:24.911707 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:24.911675 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" podUID="12e75f96-d218-400a-8b56-0cea344f3e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 00:58:25.004786 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:24.982624 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:25.159895 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:25.158374 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 00:58:25.209833 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:25.204516 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 00:58:25.209833 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:25.205107 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 00:58:25.368095 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:25.368044 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:25.433312 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:25.433258 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" podUID="1166fd98-968e-4fda-a0b9-0cfea1ed9e03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:25.625785 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:25.615686 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 30 00:58:25.883468 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:25.883379 5300 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 00:58:25.883706 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:25.883678 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" podUID="12e75f96-d218-400a-8b56-0cea344f3e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 00:58:25.898529 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:25.896049 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 00:58:25.898529 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:25.896103 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 00:58:25.898529 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:25.896136 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 00:58:25.898529 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:25.896170 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 00:58:26.857741 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:26.857706 5300 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 00:58:26.858174 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:26.857787 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" podUID="12e75f96-d218-400a-8b56-0cea344f3e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 00:58:27.007880 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:27.007832 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:27.858181 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:27.858150 5300 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 00:58:27.858674 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:27.858204 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" podUID="12e75f96-d218-400a-8b56-0cea344f3e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 00:58:28.661557 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:28.661521 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:28.858043 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:28.858010 5300 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 00:58:28.858225 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:28.858065 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" podUID="12e75f96-d218-400a-8b56-0cea344f3e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 00:58:29.862269 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:29.862237 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:29.862744 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:29.862523 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:29.863304 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:29.863286 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 00:58:29.926662 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:29.926627 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:30.341706 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:30.341652 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:31.391324 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:31.391243 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:32.106788 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:32.106741 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:33.686188 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:33.686152 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:34.435976 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:34.435930 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q"] Oct 30 00:58:34.442299 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:34.442268 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 00:58:34.469616 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:34.469586 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c104b8\"/\"distinct-w1-c104b8-kserve-self-signed-certs\"" Oct 30 00:58:34.471003 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:34.470973 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-home\") pod \"distinct-w1-c104b8-kserve-594d959c46-9n76q\" (UID: \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 00:58:34.471135 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:34.471033 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv5bg\" (UniqueName: \"kubernetes.io/projected/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-kube-api-access-lv5bg\") pod \"distinct-w1-c104b8-kserve-594d959c46-9n76q\" (UID: \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 00:58:34.471135 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:34.471126 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-tmp-dir\") pod \"distinct-w1-c104b8-kserve-594d959c46-9n76q\" (UID: \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 00:58:34.471219 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:34.471149 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-tls-certs\") pod \"distinct-w1-c104b8-kserve-594d959c46-9n76q\" (UID: \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 00:58:34.471219 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:34.471169 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-dshm\") pod \"distinct-w1-c104b8-kserve-594d959c46-9n76q\" (UID: \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 00:58:34.471361 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:34.471334 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-model-cache\") pod \"distinct-w1-c104b8-kserve-594d959c46-9n76q\" (UID: \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 00:58:34.473315 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:34.473285 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q"] Oct 30 00:58:34.479500 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:34.479475 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:34.573267 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:34.573229 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-home\") pod \"distinct-w1-c104b8-kserve-594d959c46-9n76q\" (UID: \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 00:58:34.573461 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:34.573288 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv5bg\" (UniqueName: \"kubernetes.io/projected/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-kube-api-access-lv5bg\") pod \"distinct-w1-c104b8-kserve-594d959c46-9n76q\" (UID: \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 00:58:34.573461 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:34.573322 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-tmp-dir\") pod \"distinct-w1-c104b8-kserve-594d959c46-9n76q\" (UID: \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 00:58:34.573461 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:34.573342 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-tls-certs\") pod \"distinct-w1-c104b8-kserve-594d959c46-9n76q\" (UID: \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 00:58:34.573461 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:34.573361 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-dshm\") pod \"distinct-w1-c104b8-kserve-594d959c46-9n76q\" (UID: \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 00:58:34.573461 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:34.573401 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-model-cache\") pod \"distinct-w1-c104b8-kserve-594d959c46-9n76q\" (UID: \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 00:58:34.573754 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:34.573729 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-home\") pod \"distinct-w1-c104b8-kserve-594d959c46-9n76q\" (UID: \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 00:58:34.573818 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:34.573781 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-tmp-dir\") pod \"distinct-w1-c104b8-kserve-594d959c46-9n76q\" (UID: \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 00:58:34.573900 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:34.573872 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-model-cache\") pod \"distinct-w1-c104b8-kserve-594d959c46-9n76q\" (UID: \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 00:58:34.575772 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:34.575727 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-dshm\") pod \"distinct-w1-c104b8-kserve-594d959c46-9n76q\" (UID: \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 00:58:34.575898 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:34.575881 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-tls-certs\") pod \"distinct-w1-c104b8-kserve-594d959c46-9n76q\" (UID: \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 00:58:34.592203 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:34.592175 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv5bg\" (UniqueName: \"kubernetes.io/projected/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-kube-api-access-lv5bg\") pod \"distinct-w1-c104b8-kserve-594d959c46-9n76q\" (UID: \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\") " pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 00:58:34.758796 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:34.758720 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 00:58:35.090070 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:35.090010 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q"] Oct 30 00:58:35.092679 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:58:35.092650 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4eb07a0_4a32_4091_8db0_3d0a2f9bab8b.slice/crio-6b7e4a393e22c460ddda9f8b7be32567910883c6a15aeff581fb6d6d6cdf335c WatchSource:0}: Error finding container 6b7e4a393e22c460ddda9f8b7be32567910883c6a15aeff581fb6d6d6cdf335c: Status 404 returned error can't find the container with id 6b7e4a393e22c460ddda9f8b7be32567910883c6a15aeff581fb6d6d6cdf335c Oct 30 00:58:35.139138 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:35.139101 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" event={"ID":"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b","Type":"ContainerStarted","Data":"6b7e4a393e22c460ddda9f8b7be32567910883c6a15aeff581fb6d6d6cdf335c"} Oct 30 00:58:35.500816 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:35.500755 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:36.154127 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.154088 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" event={"ID":"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b","Type":"ContainerStarted","Data":"18e1c9907ac4c4d1dcb3a430d9a8e869da48df7cd6688e69849d384f835eaa09"} Oct 30 00:58:36.193122 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.193075 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" podStartSLOduration=2.193062772 podStartE2EDuration="2.193062772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:36.192726135 +0000 UTC m=+916.508594459" watchObservedRunningTime="2026-10-30 00:58:36.193062772 +0000 UTC m=+916.508931112" Oct 30 00:58:36.435981 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.435900 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8"] Oct 30 00:58:36.439528 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.439512 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.454222 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.454200 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-fd3281-openshift-default-dockercfg-7q6pm\"" Oct 30 00:58:36.470228 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.469444 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8"] Oct 30 00:58:36.496246 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.496210 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.496450 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.496267 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db4ab11-cb3e-4351-b252-189dc15c5578-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.496450 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.496400 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.496575 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.496511 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.496707 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.496584 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.496707 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.496653 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.496707 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.496685 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.496881 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.496754 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.496881 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.496846 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktvb9\" (UniqueName: \"kubernetes.io/projected/6db4ab11-cb3e-4351-b252-189dc15c5578-kube-api-access-ktvb9\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.598271 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.598234 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktvb9\" (UniqueName: \"kubernetes.io/projected/6db4ab11-cb3e-4351-b252-189dc15c5578-kube-api-access-ktvb9\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.598451 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.598316 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.598451 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.598356 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db4ab11-cb3e-4351-b252-189dc15c5578-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.598451 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.598401 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.598614 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.598449 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.598614 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.598483 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.598614 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.598552 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.598614 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.598583 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.598856 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.598623 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.598856 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.598824 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.598974 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.598939 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.598974 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.598959 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.599115 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.599091 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.599247 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.599197 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db4ab11-cb3e-4351-b252-189dc15c5578-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.600716 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.600688 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.601018 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.601001 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.617161 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.617131 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.617291 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.617160 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktvb9\" (UniqueName: \"kubernetes.io/projected/6db4ab11-cb3e-4351-b252-189dc15c5578-kube-api-access-ktvb9\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.754012 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.753973 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:36.759249 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:36.759226 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 00:58:37.071076 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:37.070993 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8"] Oct 30 00:58:37.074505 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:58:37.074478 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db4ab11_cb3e_4351_b252_189dc15c5578.slice/crio-af646ae22948040914197e6cebae0c7fa06a30fc4a4a59be3bc06c4ce6f461e9 WatchSource:0}: Error finding container af646ae22948040914197e6cebae0c7fa06a30fc4a4a59be3bc06c4ce6f461e9: Status 404 returned error can't find the container with id af646ae22948040914197e6cebae0c7fa06a30fc4a4a59be3bc06c4ce6f461e9 Oct 30 00:58:37.077182 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:37.077148 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:58:37.077295 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:37.077217 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:58:37.077295 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:37.077248 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:58:37.163597 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:37.162956 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" event={"ID":"6db4ab11-cb3e-4351-b252-189dc15c5578","Type":"ContainerStarted","Data":"cbcdcb80484fa0373cede25f38a1e663218f7db1e8657c00c3a746f13ad60786"} Oct 30 00:58:37.163597 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:37.163020 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" event={"ID":"6db4ab11-cb3e-4351-b252-189dc15c5578","Type":"ContainerStarted","Data":"af646ae22948040914197e6cebae0c7fa06a30fc4a4a59be3bc06c4ce6f461e9"} Oct 30 00:58:37.209207 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:37.209150 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" podStartSLOduration=1.209136439 podStartE2EDuration="1.209136439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:37.204952063 +0000 UTC m=+917.520820422" watchObservedRunningTime="2026-10-30 00:58:37.209136439 +0000 UTC m=+917.525004763" Oct 30 00:58:37.260722 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:37.260666 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:37.755782 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:37.754388 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:38.757551 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:38.757508 5300 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 00:58:38.758109 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:38.757593 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" podUID="6db4ab11-cb3e-4351-b252-189dc15c5578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 00:58:39.442825 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:39.441132 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:39.755013 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:39.754979 5300 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 00:58:39.755192 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:39.755032 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" podUID="6db4ab11-cb3e-4351-b252-189dc15c5578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 00:58:40.755361 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:40.755319 5300 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 00:58:40.755815 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:40.755378 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" podUID="6db4ab11-cb3e-4351-b252-189dc15c5578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 00:58:41.171234 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:41.171194 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:41.755393 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:41.755353 5300 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 00:58:41.756014 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:41.755419 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" podUID="6db4ab11-cb3e-4351-b252-189dc15c5578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 00:58:42.755113 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:42.755077 5300 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 00:58:42.755303 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:42.755140 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" podUID="6db4ab11-cb3e-4351-b252-189dc15c5578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 00:58:42.917590 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:42.917557 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:44.554044 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:44.554006 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:44.759589 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:44.759545 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 00:58:44.773553 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:44.773521 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 00:58:44.795930 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:44.795890 5300 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:58:44.796096 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:44.795956 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" podUID="6db4ab11-cb3e-4351-b252-189dc15c5578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:58:44.967720 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:44.967632 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:44.968774 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:44.968733 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:44.969203 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:44.969185 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 00:58:45.228775 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:45.228735 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 00:58:46.130152 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:46.130116 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:47.867492 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:47.867463 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:49.627101 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:49.627057 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:49.934249 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:49.934159 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:51.378607 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:51.378574 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:51.592291 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:58:51.592254 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:58:53.029320 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:53.029277 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:54.949510 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:54.949470 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:56.636050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:56.636014 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:58.319009 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:58.318969 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:58:59.961650 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:58:59.961593 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:01.614097 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:01.614058 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:03.339430 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:03.339395 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:04.483623 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:04.483586 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:05.237641 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:05.237606 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:06.518025 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.517987 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj"] Oct 30 00:59:06.522035 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.522014 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.553623 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.553591 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj"] Oct 30 00:59:06.622314 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.622276 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.622497 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.622322 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.622497 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.622417 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.622497 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.622466 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.622622 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.622599 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6zjq\" (UniqueName: \"kubernetes.io/projected/ba5bee00-9a0a-4a43-abea-35da290cbcaa-kube-api-access-r6zjq\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.622658 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.622632 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.622658 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.622648 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.622737 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.622719 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.622798 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.622750 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.724430 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.724383 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6zjq\" (UniqueName: \"kubernetes.io/projected/ba5bee00-9a0a-4a43-abea-35da290cbcaa-kube-api-access-r6zjq\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.724430 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.724428 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.724698 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.724448 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.724698 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.724601 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.724698 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.724673 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.724882 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.724843 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.724882 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.724875 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.724997 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.724902 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.724997 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.724934 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.724997 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.724934 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.725158 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.725072 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.725406 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.725378 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.725670 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.725643 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.725819 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.725796 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.727077 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.727049 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.727455 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.727436 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.743895 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.743872 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.744856 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.744836 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6zjq\" (UniqueName: \"kubernetes.io/projected/ba5bee00-9a0a-4a43-abea-35da290cbcaa-kube-api-access-r6zjq\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:06.836597 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:06.836510 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:07.073068 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:07.066735 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:07.145427 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:07.145393 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj"] Oct 30 00:59:07.148409 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:59:07.148368 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba5bee00_9a0a_4a43_abea_35da290cbcaa.slice/crio-0ceaec7a059d78008637a5d7948568fee9fb38e31adcaaececa6b3e0332ac902 WatchSource:0}: Error finding container 0ceaec7a059d78008637a5d7948568fee9fb38e31adcaaececa6b3e0332ac902: Status 404 returned error can't find the container with id 0ceaec7a059d78008637a5d7948568fee9fb38e31adcaaececa6b3e0332ac902 Oct 30 00:59:07.151236 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:07.151188 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:59:07.151339 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:07.151274 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:59:07.151339 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:07.151310 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:59:07.382242 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:07.382198 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" event={"ID":"ba5bee00-9a0a-4a43-abea-35da290cbcaa","Type":"ContainerStarted","Data":"f4f1bbe61a07b1e4d1552f5c783cc5daef455ccc9eae727d360391c0b1c64fb9"} Oct 30 00:59:07.382430 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:07.382263 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" event={"ID":"ba5bee00-9a0a-4a43-abea-35da290cbcaa","Type":"ContainerStarted","Data":"0ceaec7a059d78008637a5d7948568fee9fb38e31adcaaececa6b3e0332ac902"} Oct 30 00:59:07.424666 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:07.424566 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" podStartSLOduration=1.424549458 podStartE2EDuration="1.424549458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:07.423801016 +0000 UTC m=+947.739669404" watchObservedRunningTime="2026-10-30 00:59:07.424549458 +0000 UTC m=+947.740417786" Oct 30 00:59:07.841785 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:07.839658 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:08.842966 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:08.842921 5300 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 00:59:08.843482 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:08.843027 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" podUID="ba5bee00-9a0a-4a43-abea-35da290cbcaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 00:59:08.843482 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:08.843341 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:09.845823 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:09.844252 5300 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 00:59:09.845823 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:09.844314 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" podUID="ba5bee00-9a0a-4a43-abea-35da290cbcaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 00:59:10.502084 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:10.502053 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:10.837265 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:10.837168 5300 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 00:59:10.837513 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:10.837483 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" podUID="ba5bee00-9a0a-4a43-abea-35da290cbcaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 00:59:11.842786 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:11.838572 5300 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 00:59:11.842786 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:11.838638 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" podUID="ba5bee00-9a0a-4a43-abea-35da290cbcaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 00:59:12.354499 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:12.354456 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:12.843202 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:12.843163 5300 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 00:59:12.844009 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:12.843224 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" podUID="ba5bee00-9a0a-4a43-abea-35da290cbcaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 00:59:13.837089 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:13.837047 5300 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 00:59:13.837257 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:13.837149 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" podUID="ba5bee00-9a0a-4a43-abea-35da290cbcaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 00:59:14.047242 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:14.047198 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:14.880634 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:14.879071 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:14.880634 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:14.880487 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:14.888153 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:14.887341 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 00:59:15.195142 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.195053 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn"] Oct 30 00:59:15.201093 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.201066 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 00:59:15.217121 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.217095 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn"] Oct 30 00:59:15.218909 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.217631 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-fd3281\"/\"kube-root-ca.crt\"" Oct 30 00:59:15.218909 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.217715 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-fd3281\"/\"openshift-service-ca.crt\"" Oct 30 00:59:15.218909 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.218112 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-fd3281\"/\"default-dockercfg-5p7xd\"" Oct 30 00:59:15.218909 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.218241 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-fd3281\"/\"distinct-2-w2-fd3281-kserve-self-signed-certs\"" Oct 30 00:59:15.234165 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.234123 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:15.331941 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.331890 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-home\") pod \"distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn\" (UID: \"3369c82d-796c-4d63-a4cd-0ea21d50124b\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 00:59:15.332119 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.332041 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-model-cache\") pod \"distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn\" (UID: \"3369c82d-796c-4d63-a4cd-0ea21d50124b\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 00:59:15.332119 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.332093 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-dshm\") pod \"distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn\" (UID: \"3369c82d-796c-4d63-a4cd-0ea21d50124b\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 00:59:15.332193 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.332147 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-tmp-dir\") pod \"distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn\" (UID: \"3369c82d-796c-4d63-a4cd-0ea21d50124b\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 00:59:15.332243 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.332214 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3369c82d-796c-4d63-a4cd-0ea21d50124b-tls-certs\") pod \"distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn\" (UID: \"3369c82d-796c-4d63-a4cd-0ea21d50124b\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 00:59:15.332297 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.332241 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84wfw\" (UniqueName: \"kubernetes.io/projected/3369c82d-796c-4d63-a4cd-0ea21d50124b-kube-api-access-84wfw\") pod \"distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn\" (UID: \"3369c82d-796c-4d63-a4cd-0ea21d50124b\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 00:59:15.434908 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.433100 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-model-cache\") pod \"distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn\" (UID: \"3369c82d-796c-4d63-a4cd-0ea21d50124b\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 00:59:15.434908 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.433162 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-dshm\") pod \"distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn\" (UID: \"3369c82d-796c-4d63-a4cd-0ea21d50124b\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 00:59:15.434908 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.433218 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-tmp-dir\") pod \"distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn\" (UID: \"3369c82d-796c-4d63-a4cd-0ea21d50124b\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 00:59:15.434908 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.433282 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3369c82d-796c-4d63-a4cd-0ea21d50124b-tls-certs\") pod \"distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn\" (UID: \"3369c82d-796c-4d63-a4cd-0ea21d50124b\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 00:59:15.434908 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.433306 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84wfw\" (UniqueName: \"kubernetes.io/projected/3369c82d-796c-4d63-a4cd-0ea21d50124b-kube-api-access-84wfw\") pod \"distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn\" (UID: \"3369c82d-796c-4d63-a4cd-0ea21d50124b\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 00:59:15.434908 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.433444 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-home\") pod \"distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn\" (UID: \"3369c82d-796c-4d63-a4cd-0ea21d50124b\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 00:59:15.434908 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.433569 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-model-cache\") pod \"distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn\" (UID: \"3369c82d-796c-4d63-a4cd-0ea21d50124b\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 00:59:15.434908 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.433860 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-home\") pod \"distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn\" (UID: \"3369c82d-796c-4d63-a4cd-0ea21d50124b\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 00:59:15.434908 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.434909 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-tmp-dir\") pod \"distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn\" (UID: \"3369c82d-796c-4d63-a4cd-0ea21d50124b\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 00:59:15.443422 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.443401 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-dshm\") pod \"distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn\" (UID: \"3369c82d-796c-4d63-a4cd-0ea21d50124b\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 00:59:15.444414 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.444364 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3369c82d-796c-4d63-a4cd-0ea21d50124b-tls-certs\") pod \"distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn\" (UID: \"3369c82d-796c-4d63-a4cd-0ea21d50124b\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 00:59:15.453274 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.453211 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84wfw\" (UniqueName: \"kubernetes.io/projected/3369c82d-796c-4d63-a4cd-0ea21d50124b-kube-api-access-84wfw\") pod \"distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn\" (UID: \"3369c82d-796c-4d63-a4cd-0ea21d50124b\") " pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 00:59:15.521842 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.521810 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 00:59:15.861231 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:15.861178 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn"] Oct 30 00:59:15.866516 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:59:15.866487 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3369c82d_796c_4d63_a4cd_0ea21d50124b.slice/crio-09dc9a4dc98755474a9de6c17772a22602b8d7e3e869efaaf3c09ddf13f47643 WatchSource:0}: Error finding container 09dc9a4dc98755474a9de6c17772a22602b8d7e3e869efaaf3c09ddf13f47643: Status 404 returned error can't find the container with id 09dc9a4dc98755474a9de6c17772a22602b8d7e3e869efaaf3c09ddf13f47643 Oct 30 00:59:16.089083 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:16.089045 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:16.449343 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:16.449296 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" event={"ID":"3369c82d-796c-4d63-a4cd-0ea21d50124b","Type":"ContainerStarted","Data":"ec43d364f1d122ffcfb274c18867ad0763d34f21317b7b0fded794b4e7189f36"} Oct 30 00:59:16.449732 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:16.449356 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" event={"ID":"3369c82d-796c-4d63-a4cd-0ea21d50124b","Type":"ContainerStarted","Data":"09dc9a4dc98755474a9de6c17772a22602b8d7e3e869efaaf3c09ddf13f47643"} Oct 30 00:59:16.488806 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:16.488736 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" podStartSLOduration=1.488722476 podStartE2EDuration="1.488722476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:16.486662371 +0000 UTC m=+956.802530696" watchObservedRunningTime="2026-10-30 00:59:16.488722476 +0000 UTC m=+956.804590800" Oct 30 00:59:16.522228 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:16.522198 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 00:59:17.750457 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:17.750421 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:19.392831 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:19.392801 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:21.141506 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:21.141466 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:21.593145 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:59:21.593108 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:59:22.773588 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:22.773543 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:24.424318 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:24.424285 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:25.522500 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:25.522463 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 00:59:25.535821 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:25.535793 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 00:59:26.148211 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:26.148180 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:26.519149 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:26.519121 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 00:59:26.990658 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:26.990624 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:28.153716 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:28.153685 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:29.861608 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:29.861574 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:30.073631 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:30.073596 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:31.680116 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:31.680083 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:33.338938 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:33.338905 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:34.887929 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:34.887882 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:36.198847 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.198817 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd"] Oct 30 00:59:36.210994 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.210965 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.211422 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.211393 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4"] Oct 30 00:59:36.220846 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.216026 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 00:59:36.233943 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.233912 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd"] Oct 30 00:59:36.242396 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.242363 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4"] Oct 30 00:59:36.244042 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.244005 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-c4847c\"/\"distinct-2-w6-c4847c-kserve-self-signed-certs\"" Oct 30 00:59:36.259491 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.259443 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:36.300063 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.300028 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.300063 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.300066 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9699f\" (UniqueName: \"kubernetes.io/projected/d1f758e7-acc5-4a72-84cc-c70e23841347-kube-api-access-9699f\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.300290 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.300177 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-dshm\") pod \"distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4\" (UID: \"a36b0810-b564-4c4f-900f-6ec73fabd405\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 00:59:36.300290 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.300238 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.300290 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.300258 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.300447 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.300294 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.300447 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.300357 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-tmp-dir\") pod \"distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4\" (UID: \"a36b0810-b564-4c4f-900f-6ec73fabd405\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 00:59:36.300512 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.300477 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1f758e7-acc5-4a72-84cc-c70e23841347-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.300512 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.300498 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.300597 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.300516 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.300597 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.300538 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8sfq\" (UniqueName: \"kubernetes.io/projected/a36b0810-b564-4c4f-900f-6ec73fabd405-kube-api-access-t8sfq\") pod \"distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4\" (UID: \"a36b0810-b564-4c4f-900f-6ec73fabd405\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 00:59:36.300674 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.300639 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.300721 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.300676 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a36b0810-b564-4c4f-900f-6ec73fabd405-tls-certs\") pod \"distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4\" (UID: \"a36b0810-b564-4c4f-900f-6ec73fabd405\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 00:59:36.300721 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.300705 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-model-cache\") pod \"distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4\" (UID: \"a36b0810-b564-4c4f-900f-6ec73fabd405\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 00:59:36.300847 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.300825 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-home\") pod \"distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4\" (UID: \"a36b0810-b564-4c4f-900f-6ec73fabd405\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 00:59:36.347189 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.347159 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2"] Oct 30 00:59:36.351061 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.351040 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.375828 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.375798 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2"] Oct 30 00:59:36.401797 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.401742 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-model-cache\") pod \"distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4\" (UID: \"a36b0810-b564-4c4f-900f-6ec73fabd405\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 00:59:36.401982 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.401827 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-home\") pod \"distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4\" (UID: \"a36b0810-b564-4c4f-900f-6ec73fabd405\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 00:59:36.401982 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.401867 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.401982 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.401908 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.401982 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.401939 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9699f\" (UniqueName: \"kubernetes.io/projected/d1f758e7-acc5-4a72-84cc-c70e23841347-kube-api-access-9699f\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.402150 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.402018 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-dshm\") pod \"distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4\" (UID: \"a36b0810-b564-4c4f-900f-6ec73fabd405\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 00:59:36.402150 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.402056 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.402150 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.402114 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.408906 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.402173 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.408906 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.402228 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.408906 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.402257 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vdrx\" (UniqueName: \"kubernetes.io/projected/59536065-cfa0-4f0c-a2ca-55284c369d81-kube-api-access-7vdrx\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.408906 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.402292 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.408906 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.402333 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.408906 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.402360 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.408906 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.402382 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-tmp-dir\") pod \"distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4\" (UID: \"a36b0810-b564-4c4f-900f-6ec73fabd405\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 00:59:36.408906 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.402420 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1f758e7-acc5-4a72-84cc-c70e23841347-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.408906 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.402442 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.408906 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.402471 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.408906 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.402495 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.408906 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.402529 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8sfq\" (UniqueName: \"kubernetes.io/projected/a36b0810-b564-4c4f-900f-6ec73fabd405-kube-api-access-t8sfq\") pod \"distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4\" (UID: \"a36b0810-b564-4c4f-900f-6ec73fabd405\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 00:59:36.408906 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.402577 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.408906 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.402599 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.408906 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.402626 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59536065-cfa0-4f0c-a2ca-55284c369d81-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.408906 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.402662 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a36b0810-b564-4c4f-900f-6ec73fabd405-tls-certs\") pod \"distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4\" (UID: \"a36b0810-b564-4c4f-900f-6ec73fabd405\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 00:59:36.424620 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.424589 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1f758e7-acc5-4a72-84cc-c70e23841347-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.430907 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.425504 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a36b0810-b564-4c4f-900f-6ec73fabd405-tls-certs\") pod \"distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4\" (UID: \"a36b0810-b564-4c4f-900f-6ec73fabd405\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 00:59:36.431135 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.425838 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.431553 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.431524 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.432107 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.432083 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.433639 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.433610 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.433944 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.433921 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-model-cache\") pod \"distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4\" (UID: \"a36b0810-b564-4c4f-900f-6ec73fabd405\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 00:59:36.434265 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.434160 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-home\") pod \"distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4\" (UID: \"a36b0810-b564-4c4f-900f-6ec73fabd405\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 00:59:36.434603 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.434579 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-tmp-dir\") pod \"distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4\" (UID: \"a36b0810-b564-4c4f-900f-6ec73fabd405\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 00:59:36.438520 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.436748 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.442428 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.442403 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.459427 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.458945 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-dshm\") pod \"distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4\" (UID: \"a36b0810-b564-4c4f-900f-6ec73fabd405\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 00:59:36.483059 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.482843 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.487328 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.487289 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9699f\" (UniqueName: \"kubernetes.io/projected/d1f758e7-acc5-4a72-84cc-c70e23841347-kube-api-access-9699f\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.488071 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.488049 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8sfq\" (UniqueName: \"kubernetes.io/projected/a36b0810-b564-4c4f-900f-6ec73fabd405-kube-api-access-t8sfq\") pod \"distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4\" (UID: \"a36b0810-b564-4c4f-900f-6ec73fabd405\") " pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 00:59:36.504206 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.504177 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.504362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.504258 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.504362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.504295 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.504362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.504313 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.504362 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.504337 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vdrx\" (UniqueName: \"kubernetes.io/projected/59536065-cfa0-4f0c-a2ca-55284c369d81-kube-api-access-7vdrx\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.504572 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.504372 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.504572 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.504401 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.504572 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.504464 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.504572 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.504488 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59536065-cfa0-4f0c-a2ca-55284c369d81-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.504846 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.504695 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.504999 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.504973 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.505180 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.504983 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.505293 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.505104 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.505293 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.505221 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59536065-cfa0-4f0c-a2ca-55284c369d81-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.506443 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.506422 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.506973 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.506953 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.525365 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.525335 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.525463 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.525404 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vdrx\" (UniqueName: \"kubernetes.io/projected/59536065-cfa0-4f0c-a2ca-55284c369d81-kube-api-access-7vdrx\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.538050 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.538026 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:36.550670 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.550633 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 00:59:36.666040 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.666007 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:36.746901 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.746869 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:36.855949 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.855920 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd"] Oct 30 00:59:36.858953 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:59:36.858918 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f758e7_acc5_4a72_84cc_c70e23841347.slice/crio-0b218986550c01219dfab921a253031f96f8b52178d54ae5ae375d77e80be7d7 WatchSource:0}: Error finding container 0b218986550c01219dfab921a253031f96f8b52178d54ae5ae375d77e80be7d7: Status 404 returned error can't find the container with id 0b218986550c01219dfab921a253031f96f8b52178d54ae5ae375d77e80be7d7 Oct 30 00:59:36.861585 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.861551 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:59:36.861686 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.861619 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:59:36.861686 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.861650 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:59:36.869749 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.869719 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4"] Oct 30 00:59:36.875716 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:59:36.875693 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda36b0810_b564_4c4f_900f_6ec73fabd405.slice/crio-0036fc8b6874ab253986a1844e418b919b35e71a75f15889a5993b261b545a5c WatchSource:0}: Error finding container 0036fc8b6874ab253986a1844e418b919b35e71a75f15889a5993b261b545a5c: Status 404 returned error can't find the container with id 0036fc8b6874ab253986a1844e418b919b35e71a75f15889a5993b261b545a5c Oct 30 00:59:36.969351 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.969261 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2"] Oct 30 00:59:36.972581 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:59:36.972554 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59536065_cfa0_4f0c_a2ca_55284c369d81.slice/crio-19307bb9bb200a86d3bef422e2c602fa6abe6425f04e74d0b82d65792123d95a WatchSource:0}: Error finding container 19307bb9bb200a86d3bef422e2c602fa6abe6425f04e74d0b82d65792123d95a: Status 404 returned error can't find the container with id 19307bb9bb200a86d3bef422e2c602fa6abe6425f04e74d0b82d65792123d95a Oct 30 00:59:36.975976 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.975936 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:59:36.976093 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.976049 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:59:36.976154 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:36.976095 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:59:37.587860 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:37.587819 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" event={"ID":"d1f758e7-acc5-4a72-84cc-c70e23841347","Type":"ContainerStarted","Data":"6010d7ae1e66318d7251f66cab044d4c267ebdbfc049347df134d4565c727f89"} Oct 30 00:59:37.588372 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:37.588352 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" event={"ID":"d1f758e7-acc5-4a72-84cc-c70e23841347","Type":"ContainerStarted","Data":"0b218986550c01219dfab921a253031f96f8b52178d54ae5ae375d77e80be7d7"} Oct 30 00:59:37.591392 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:37.591362 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" event={"ID":"a36b0810-b564-4c4f-900f-6ec73fabd405","Type":"ContainerStarted","Data":"baf22d4273bdbf376eda851e14ea5ce738e7021ef495d6b332aa8121f113d245"} Oct 30 00:59:37.591944 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:37.591912 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" event={"ID":"a36b0810-b564-4c4f-900f-6ec73fabd405","Type":"ContainerStarted","Data":"0036fc8b6874ab253986a1844e418b919b35e71a75f15889a5993b261b545a5c"} Oct 30 00:59:37.593867 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:37.593815 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" event={"ID":"59536065-cfa0-4f0c-a2ca-55284c369d81","Type":"ContainerStarted","Data":"42661b1ed5c2d29b94d756ac672ebbaf0990f6d22d30b33c09dbf42ddf99c06f"} Oct 30 00:59:37.593977 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:37.593863 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" event={"ID":"59536065-cfa0-4f0c-a2ca-55284c369d81","Type":"ContainerStarted","Data":"19307bb9bb200a86d3bef422e2c602fa6abe6425f04e74d0b82d65792123d95a"} Oct 30 00:59:37.669786 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:37.668855 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:37.698467 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:37.698205 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" podStartSLOduration=1.698185729 podStartE2EDuration="1.698185729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:37.696441632 +0000 UTC m=+978.012309957" watchObservedRunningTime="2026-10-30 00:59:37.698185729 +0000 UTC m=+978.014054059" Oct 30 00:59:37.700257 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:37.700205 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" podStartSLOduration=1.700191413 podStartE2EDuration="1.700191413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:37.656120355 +0000 UTC m=+977.971988684" watchObservedRunningTime="2026-10-30 00:59:37.700191413 +0000 UTC m=+978.016059792" Oct 30 00:59:37.753785 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:37.753303 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" podStartSLOduration=1.7532823789999998 podStartE2EDuration="1.753282379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:37.738954609 +0000 UTC m=+978.054822934" watchObservedRunningTime="2026-10-30 00:59:37.753282379 +0000 UTC m=+978.069150707" Oct 30 00:59:38.549784 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:38.547877 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:38.549784 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:38.548037 5300 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 00:59:38.549784 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:38.548082 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" podUID="d1f758e7-acc5-4a72-84cc-c70e23841347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 00:59:38.553783 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:38.552839 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 00:59:38.641787 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:38.641220 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:38.670264 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:38.670227 5300 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 00:59:38.670418 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:38.670286 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" podUID="59536065-cfa0-4f0c-a2ca-55284c369d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 00:59:39.542795 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:39.542156 5300 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 00:59:39.542795 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:39.542218 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" podUID="d1f758e7-acc5-4a72-84cc-c70e23841347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 00:59:39.671786 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:39.671132 5300 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 00:59:39.671786 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:39.671191 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" podUID="59536065-cfa0-4f0c-a2ca-55284c369d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 00:59:40.540785 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:40.540133 5300 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 00:59:40.540785 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:40.540197 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" podUID="d1f758e7-acc5-4a72-84cc-c70e23841347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 00:59:40.667782 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:40.667403 5300 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 00:59:40.667782 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:40.667485 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" podUID="59536065-cfa0-4f0c-a2ca-55284c369d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 00:59:40.814290 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:40.814200 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:41.542784 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:41.542129 5300 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 00:59:41.542784 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:41.542214 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" podUID="d1f758e7-acc5-4a72-84cc-c70e23841347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 00:59:41.669203 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:41.668794 5300 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 00:59:41.669203 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:41.668874 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" podUID="59536065-cfa0-4f0c-a2ca-55284c369d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 00:59:42.553785 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:42.552884 5300 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 00:59:42.553785 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:42.552954 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" podUID="d1f758e7-acc5-4a72-84cc-c70e23841347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 00:59:42.667786 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:42.665589 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:42.674894 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:42.674299 5300 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 00:59:42.674894 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:42.674356 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" podUID="59536065-cfa0-4f0c-a2ca-55284c369d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 00:59:43.538459 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:43.538417 5300 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 00:59:43.538643 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:43.538487 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" podUID="d1f758e7-acc5-4a72-84cc-c70e23841347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 00:59:43.667206 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:43.667168 5300 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 00:59:43.667848 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:43.667227 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" podUID="59536065-cfa0-4f0c-a2ca-55284c369d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 00:59:44.367561 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:44.367492 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:44.672188 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:44.672118 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:44.672596 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:44.672549 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:44.673364 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:44.673343 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 00:59:45.581926 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:45.581887 5300 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 00:59:45.582148 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:45.581947 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" podUID="d1f758e7-acc5-4a72-84cc-c70e23841347" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 00:59:45.809437 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:45.809395 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:45.809900 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:45.809792 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:45.810821 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:45.810797 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 00:59:46.205511 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:46.205458 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:46.551269 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:46.551229 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 00:59:46.564141 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:46.564110 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 00:59:46.717877 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:46.717843 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 00:59:47.878513 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:47.878400 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:49.467525 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:49.467480 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:51.133547 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.133510 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:51.248890 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.248855 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5"] Oct 30 00:59:51.254889 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.254863 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.289271 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.289231 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.289448 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.289282 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.289448 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.289314 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/443d5afc-4682-442b-9dbb-9e125c2973a0-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.289448 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.289432 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.289588 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.289464 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrbd2\" (UniqueName: \"kubernetes.io/projected/443d5afc-4682-442b-9dbb-9e125c2973a0-kube-api-access-rrbd2\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.289588 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.289527 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.289588 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.289560 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.289811 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.289623 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.289811 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.289655 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.291816 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.291786 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5"] Oct 30 00:59:51.390852 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.390743 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrbd2\" (UniqueName: \"kubernetes.io/projected/443d5afc-4682-442b-9dbb-9e125c2973a0-kube-api-access-rrbd2\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.390852 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.390832 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.391079 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.390868 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.391079 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.390929 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.391079 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.390958 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.391235 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.391083 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.391235 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.391110 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.391235 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.391138 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/443d5afc-4682-442b-9dbb-9e125c2973a0-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.391384 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.391261 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.391384 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.391273 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-workload-certs\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.391384 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.391289 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-data\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.391543 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.391378 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-workload-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.391597 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.391554 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-credential-socket\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.392176 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.392150 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/443d5afc-4682-442b-9dbb-9e125c2973a0-istiod-ca-cert\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.393694 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.393671 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-envoy\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.393932 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.393898 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-podinfo\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.415740 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.415703 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-token\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.416538 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.416509 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrbd2\" (UniqueName: \"kubernetes.io/projected/443d5afc-4682-442b-9dbb-9e125c2973a0-kube-api-access-rrbd2\") pod \"e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.569714 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.569684 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:51.597291 ip-10-0-140-206 kubenswrapper[5300]: E1030 00:59:51.597247 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 00:59:51.740226 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.740180 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2"] Oct 30 00:59:51.746336 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.746308 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:51.787975 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.787940 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2"] Oct 30 00:59:51.797135 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.797101 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:51.797135 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.797140 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:51.797335 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.797168 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:51.797335 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.797320 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:51.797402 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.797351 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:51.797447 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.797430 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:51.797489 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.797475 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r8ln\" (UniqueName: \"kubernetes.io/projected/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-kube-api-access-7r8ln\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:51.797557 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.797545 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:51.797620 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.797610 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:51.883901 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.883875 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5"] Oct 30 00:59:51.886303 ip-10-0-140-206 kubenswrapper[5300]: W1030 00:59:51.886277 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod443d5afc_4682_442b_9dbb_9e125c2973a0.slice/crio-1a2c2168656980714b2cceff19d44c6c1af0cc984e5a0b4de9a267dabca45121 WatchSource:0}: Error finding container 1a2c2168656980714b2cceff19d44c6c1af0cc984e5a0b4de9a267dabca45121: Status 404 returned error can't find the container with id 1a2c2168656980714b2cceff19d44c6c1af0cc984e5a0b4de9a267dabca45121 Oct 30 00:59:51.889172 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.889138 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:59:51.889297 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.889216 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:59:51.889297 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.889265 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:59:51.898409 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.898387 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:51.898527 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.898417 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:51.898527 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.898464 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:51.898527 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.898485 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r8ln\" (UniqueName: \"kubernetes.io/projected/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-kube-api-access-7r8ln\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:51.898687 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.898577 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:51.898739 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.898699 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:51.898851 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.898831 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:51.898910 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.898882 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:51.898910 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.898886 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-data\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:51.899017 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.898906 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-credential-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:51.899017 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.898938 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:51.899017 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.898987 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-workload-certs\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:51.899453 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.899432 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-workload-socket\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:51.899531 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.899515 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istiod-ca-cert\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:51.901360 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.901336 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-envoy\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:51.901737 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.901717 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-podinfo\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:51.917889 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.917858 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-token\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:51.918487 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:51.918297 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r8ln\" (UniqueName: \"kubernetes.io/projected/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-kube-api-access-7r8ln\") pod \"e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:52.061139 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:52.061094 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:52.428977 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:52.428807 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2"] Oct 30 00:59:52.477504 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:52.477454 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:59:52.477656 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:52.477551 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:59:52.477656 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:52.477593 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 00:59:52.810898 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:52.808727 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" event={"ID":"443d5afc-4682-442b-9dbb-9e125c2973a0","Type":"ContainerStarted","Data":"ed8defd10942afc601c1cd69f3b790fa972e49ad971bf514202ad1920f8075dd"} Oct 30 00:59:52.810898 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:52.808807 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" event={"ID":"443d5afc-4682-442b-9dbb-9e125c2973a0","Type":"ContainerStarted","Data":"1a2c2168656980714b2cceff19d44c6c1af0cc984e5a0b4de9a267dabca45121"} Oct 30 00:59:52.812198 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:52.812161 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" event={"ID":"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1","Type":"ContainerStarted","Data":"b9c48d57f3af691089f54b17ef3068e89ce40d0bcfc69bafd170ef61c6892009"} Oct 30 00:59:52.812385 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:52.812370 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" event={"ID":"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1","Type":"ContainerStarted","Data":"53468c57ccb3e9f8e0926de89151075512676964ff2fddb9fb9a7dd3ae3c0061"} Oct 30 00:59:52.886785 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:52.886340 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" podStartSLOduration=1.886317727 podStartE2EDuration="1.886317727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:52.864049756 +0000 UTC m=+993.179918092" watchObservedRunningTime="2026-10-30 00:59:52.886317727 +0000 UTC m=+993.202186051" Oct 30 00:59:52.934781 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:52.934420 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" podStartSLOduration=1.934398305 podStartE2EDuration="1.934398305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:52.919150882 +0000 UTC m=+993.235019244" watchObservedRunningTime="2026-10-30 00:59:52.934398305 +0000 UTC m=+993.250266632" Oct 30 00:59:53.063972 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:53.063816 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 00:59:53.079068 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:53.078945 5300 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 00:59:53.079068 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:53.079025 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" podUID="8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 00:59:53.573599 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:53.573563 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 00:59:53.574289 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:53.574264 5300 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 00:59:53.574452 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:53.574421 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" podUID="443d5afc-4682-442b-9dbb-9e125c2973a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 00:59:53.600335 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:53.600303 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:54.063643 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:54.063185 5300 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 00:59:54.063643 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:54.063249 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" podUID="8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 00:59:54.167850 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:54.166854 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:59:54.232478 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:54.231477 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" podUID="d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 00:59:54.394957 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:54.394852 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:59:54.445656 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:54.444952 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" podUID="1166fd98-968e-4fda-a0b9-0cfea1ed9e03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:59:54.522937 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:54.522891 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:59:54.577202 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:54.577163 5300 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 00:59:54.577799 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:54.577739 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" podUID="443d5afc-4682-442b-9dbb-9e125c2973a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 00:59:54.708938 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:54.707856 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:59:54.897785 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:54.897048 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 00:59:54.902794 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:54.898278 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 00:59:55.066810 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:55.064183 5300 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 00:59:55.066810 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:55.064245 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" podUID="8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 00:59:55.571253 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:55.571216 5300 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 00:59:55.571538 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:55.571509 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" podUID="443d5afc-4682-442b-9dbb-9e125c2973a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 00:59:55.612280 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:55.612185 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:56.076651 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:56.076614 5300 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 00:59:56.076962 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:56.076928 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" podUID="8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 00:59:56.576020 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:56.573534 5300 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 00:59:56.576276 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:56.576248 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" podUID="443d5afc-4682-442b-9dbb-9e125c2973a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 00:59:57.067786 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:57.067497 5300 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 00:59:57.067786 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:57.067572 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" podUID="8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 00:59:57.572428 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:57.572389 5300 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 00:59:57.572737 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:57.572706 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" podUID="443d5afc-4682-442b-9dbb-9e125c2973a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 00:59:57.694114 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:57.694081 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:58.062949 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:58.062902 5300 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 00:59:58.063230 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:58.063185 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" podUID="8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 00:59:58.575129 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:58.572229 5300 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 00:59:58.575129 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:58.572287 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" podUID="443d5afc-4682-442b-9dbb-9e125c2973a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 00:59:59.062367 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:59.062330 5300 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 00:59:59.062571 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:59.062387 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" podUID="8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 00:59:59.551216 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:59.551178 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 00:59:59.570486 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:59.570453 5300 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 00:59:59.570639 ip-10-0-140-206 kubenswrapper[5300]: I1030 00:59:59.570507 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" podUID="443d5afc-4682-442b-9dbb-9e125c2973a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 01:00:00.061883 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:00.061837 5300 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 01:00:00.062282 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:00.061912 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" podUID="8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 01:00:00.168603 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:00.168561 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c"] Oct 30 01:00:00.175609 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:00.175584 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c" Oct 30 01:00:00.220349 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:00.219385 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c"] Oct 30 01:00:00.274990 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:00.274938 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\"" Oct 30 01:00:00.324315 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:00.324217 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9e65cfb-4f44-417e-b378-f6f87da940f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c\" (UID: \"e9e65cfb-4f44-417e-b378-f6f87da940f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c" Oct 30 01:00:00.324315 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:00.324266 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-967xx\" (UniqueName: \"kubernetes.io/projected/e9e65cfb-4f44-417e-b378-f6f87da940f5-kube-api-access-967xx\") pod \"maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c\" (UID: \"e9e65cfb-4f44-417e-b378-f6f87da940f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c" Oct 30 01:00:00.425521 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:00.425480 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9e65cfb-4f44-417e-b378-f6f87da940f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c\" (UID: \"e9e65cfb-4f44-417e-b378-f6f87da940f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c" Oct 30 01:00:00.425699 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:00.425532 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-967xx\" (UniqueName: \"kubernetes.io/projected/e9e65cfb-4f44-417e-b378-f6f87da940f5-kube-api-access-967xx\") pod \"maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c\" (UID: \"e9e65cfb-4f44-417e-b378-f6f87da940f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c" Oct 30 01:00:00.426196 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:00.426170 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9e65cfb-4f44-417e-b378-f6f87da940f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c\" (UID: \"e9e65cfb-4f44-417e-b378-f6f87da940f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c" Oct 30 01:00:00.447043 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:00.447010 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-967xx\" (UniqueName: \"kubernetes.io/projected/e9e65cfb-4f44-417e-b378-f6f87da940f5-kube-api-access-967xx\") pod \"maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c\" (UID: \"e9e65cfb-4f44-417e-b378-f6f87da940f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c" Oct 30 01:00:00.528607 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:00.528575 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c" Oct 30 01:00:00.575779 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:00.575679 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 01:00:00.576461 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:00.576434 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 01:00:00.577407 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:00.577383 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 01:00:00.835891 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:00.835740 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c"] Oct 30 01:00:00.971987 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:00.971949 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c" event={"ID":"e9e65cfb-4f44-417e-b378-f6f87da940f5","Type":"ContainerStarted","Data":"a9c87456ce8658ce8387e772804e8684ab40f4f01116ff08e5d18a2f13cd3d6b"} Oct 30 01:00:01.065735 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:01.065708 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 01:00:01.066283 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:01.066263 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 01:00:01.066906 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:01.066889 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 01:00:01.391152 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:01.391106 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:03.205702 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:03.205673 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:04.846595 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:04.846557 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:05.001174 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:05.000932 5300 generic.go:356] "Generic (PLEG): container finished" podID="e9e65cfb-4f44-417e-b378-f6f87da940f5" containerID="f1489895597afef631fd640b281918d90fb700e82ab3f7ab25c27014e79b7cd1" exitCode=60 Oct 30 01:00:05.001174 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:05.001077 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c" event={"ID":"e9e65cfb-4f44-417e-b378-f6f87da940f5","Type":"ContainerDied","Data":"f1489895597afef631fd640b281918d90fb700e82ab3f7ab25c27014e79b7cd1"} Oct 30 01:00:05.001888 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:05.001837 5300 scope.go:122] "RemoveContainer" containerID="f1489895597afef631fd640b281918d90fb700e82ab3f7ab25c27014e79b7cd1" Oct 30 01:00:05.960298 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:05.960266 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9"] Oct 30 01:00:05.964381 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:05.964361 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.003864 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.003832 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9"] Oct 30 01:00:06.016964 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.016914 5300 generic.go:356] "Generic (PLEG): container finished" podID="e9e65cfb-4f44-417e-b378-f6f87da940f5" containerID="d11d9fb8fbd7d15d5c06da312e136bfde43970e87687634c3bc37da8cccd7dde" exitCode=60 Oct 30 01:00:06.016964 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.016943 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c" event={"ID":"e9e65cfb-4f44-417e-b378-f6f87da940f5","Type":"ContainerDied","Data":"d11d9fb8fbd7d15d5c06da312e136bfde43970e87687634c3bc37da8cccd7dde"} Oct 30 01:00:06.017200 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.017017 5300 scope.go:122] "RemoveContainer" containerID="f1489895597afef631fd640b281918d90fb700e82ab3f7ab25c27014e79b7cd1" Oct 30 01:00:06.017592 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.017574 5300 scope.go:122] "RemoveContainer" containerID="d11d9fb8fbd7d15d5c06da312e136bfde43970e87687634c3bc37da8cccd7dde" Oct 30 01:00:06.017945 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:00:06.017917 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c_odh-ai-gateway-infra(e9e65cfb-4f44-417e-b378-f6f87da940f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c" podUID="e9e65cfb-4f44-417e-b378-f6f87da940f5" Oct 30 01:00:06.106417 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.106382 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.106585 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.106452 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.106627 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.106584 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.106670 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.106647 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e69d76c4-782a-4518-b52a-72c1cfd7797b-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.106727 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.106708 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.106866 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.106849 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzdrh\" (UniqueName: \"kubernetes.io/projected/e69d76c4-782a-4518-b52a-72c1cfd7797b-kube-api-access-xzdrh\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.106969 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.106955 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.107087 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.107064 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.107180 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.107160 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.208188 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.208098 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.208188 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.208142 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.208188 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.208177 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.208460 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.208235 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.208460 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.208296 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.208460 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.208356 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.208460 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.208407 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e69d76c4-782a-4518-b52a-72c1cfd7797b-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.208460 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.208440 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.208713 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.208575 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.208713 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.208586 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.208849 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.208792 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.208849 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.208802 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.209147 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.209124 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e69d76c4-782a-4518-b52a-72c1cfd7797b-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.209226 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.209129 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzdrh\" (UniqueName: \"kubernetes.io/projected/e69d76c4-782a-4518-b52a-72c1cfd7797b-kube-api-access-xzdrh\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.210774 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.210737 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.210946 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.210928 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.228939 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.228911 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.229270 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.229251 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzdrh\" (UniqueName: \"kubernetes.io/projected/e69d76c4-782a-4518-b52a-72c1cfd7797b-kube-api-access-xzdrh\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.316278 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.316243 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:06.509349 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.509311 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:06.633262 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.633222 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9"] Oct 30 01:00:06.634561 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:00:06.634524 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69d76c4_782a_4518_b52a_72c1cfd7797b.slice/crio-78149690b555088140e06105f80c616b4da93cb8bd78b0a024a5793e711e92ce WatchSource:0}: Error finding container 78149690b555088140e06105f80c616b4da93cb8bd78b0a024a5793e711e92ce: Status 404 returned error can't find the container with id 78149690b555088140e06105f80c616b4da93cb8bd78b0a024a5793e711e92ce Oct 30 01:00:06.637096 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.637061 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:00:06.637178 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.637150 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:00:06.637214 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:06.637199 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:00:07.025702 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:07.025672 5300 scope.go:122] "RemoveContainer" containerID="d11d9fb8fbd7d15d5c06da312e136bfde43970e87687634c3bc37da8cccd7dde" Oct 30 01:00:07.026328 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:00:07.026291 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c_odh-ai-gateway-infra(e9e65cfb-4f44-417e-b378-f6f87da940f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c" podUID="e9e65cfb-4f44-417e-b378-f6f87da940f5" Oct 30 01:00:07.027233 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:07.027210 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" event={"ID":"e69d76c4-782a-4518-b52a-72c1cfd7797b","Type":"ContainerStarted","Data":"43c2d13d41d5d2f7e8e3510884705fea3de0052fdb4b7998943f729e38699338"} Oct 30 01:00:07.027337 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:07.027252 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" event={"ID":"e69d76c4-782a-4518-b52a-72c1cfd7797b","Type":"ContainerStarted","Data":"78149690b555088140e06105f80c616b4da93cb8bd78b0a024a5793e711e92ce"} Oct 30 01:00:07.110575 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:07.110499 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" podStartSLOduration=2.110484756 podStartE2EDuration="2.110484756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:07.109804053 +0000 UTC m=+1007.425672399" watchObservedRunningTime="2026-10-30 01:00:07.110484756 +0000 UTC m=+1007.426353080" Oct 30 01:00:07.316729 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:07.316650 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:07.323815 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:07.318699 5300 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:00:07.323815 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:07.318785 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" podUID="e69d76c4-782a-4518-b52a-72c1cfd7797b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 01:00:08.317422 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:08.317389 5300 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:00:08.317867 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:08.317445 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" podUID="e69d76c4-782a-4518-b52a-72c1cfd7797b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 01:00:08.471700 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:08.471656 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:09.317390 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:09.317356 5300 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:00:09.317567 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:09.317410 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" podUID="e69d76c4-782a-4518-b52a-72c1cfd7797b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 01:00:10.144445 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:10.144404 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:10.317521 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:10.317485 5300 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:00:10.317699 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:10.317543 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" podUID="e69d76c4-782a-4518-b52a-72c1cfd7797b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 01:00:11.317756 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:11.317723 5300 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:00:11.318400 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:11.318346 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" podUID="e69d76c4-782a-4518-b52a-72c1cfd7797b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 01:00:12.332025 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:12.331984 5300 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:00:12.332645 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:12.332611 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" podUID="e69d76c4-782a-4518-b52a-72c1cfd7797b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 01:00:12.513934 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:12.513881 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:12.712960 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:12.712853 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 01:00:12.896679 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:12.895914 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:12.898015 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:12.897981 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:13.208995 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:13.208928 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:13.248584 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:13.248529 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" podUID="d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:13.332693 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:13.319179 5300 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:00:13.332693 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:13.319243 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" podUID="e69d76c4-782a-4518-b52a-72c1cfd7797b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 01:00:13.492815 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:13.414221 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:13.492815 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:13.441567 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" podUID="1166fd98-968e-4fda-a0b9-0cfea1ed9e03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:13.548779 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:13.527268 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:13.548779 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:13.533813 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 01:00:13.752840 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:13.748279 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 01:00:13.752840 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:13.749354 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 01:00:13.899988 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:13.899945 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 30 01:00:13.926963 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:13.926741 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:14.221348 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:14.221292 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:14.248184 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:14.248145 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:14.250039 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:14.249988 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" podUID="d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:14.340715 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:14.340663 5300 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:00:14.341351 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:14.341308 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" podUID="e69d76c4-782a-4518-b52a-72c1cfd7797b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 01:00:14.352823 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:14.352797 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 01:00:14.353434 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:14.353416 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 01:00:14.354473 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:14.354457 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 01:00:14.354647 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:14.354632 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 01:00:14.439806 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:14.423905 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:14.439806 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:14.424550 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 01:00:14.452836 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:14.441884 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" podUID="1166fd98-968e-4fda-a0b9-0cfea1ed9e03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:14.452836 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:14.447311 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 01:00:14.539845 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:14.537860 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:14.662791 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:14.645862 5300 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:00:14.662791 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:14.645922 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" podUID="e7d2e040-342a-4ea4-a23d-c36722f2d64e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:14.807930 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:14.805988 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 01:00:14.904140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:14.904100 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:14.964786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:14.955056 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:15.026981 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:15.026954 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 01:00:15.114189 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:15.114106 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 01:00:15.285785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:15.276842 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 01:00:15.285785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:15.276903 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" podUID="d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 01:00:15.438783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:15.438679 5300 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:00:15.439520 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:15.439486 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" podUID="e69d76c4-782a-4518-b52a-72c1cfd7797b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 01:00:15.460935 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:15.460887 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:15.505065 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:15.505033 5300 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:00:15.512891 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:15.512853 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" podUID="db62d92e-f4b6-480b-9545-569baf9670a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:15.538423 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:15.538368 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:15.614494 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:15.538490 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 01:00:15.620858 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:15.620832 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 01:00:15.631915 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:15.626227 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 01:00:15.631915 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:15.628014 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 01:00:15.791583 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:15.791549 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 01:00:15.800957 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:15.800921 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 01:00:16.401004 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:16.400973 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:16.401624 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:16.401597 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:16.402859 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:16.402726 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:00:16.508138 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:16.508098 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:18.258565 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:18.258529 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:19.903632 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:19.903591 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:20.813924 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:20.813898 5300 scope.go:122] "RemoveContainer" containerID="d11d9fb8fbd7d15d5c06da312e136bfde43970e87687634c3bc37da8cccd7dde" Oct 30 01:00:21.461035 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.460952 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm"] Oct 30 01:00:21.464995 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.464975 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.502127 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.502094 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm"] Oct 30 01:00:21.586173 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.585603 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68f9be1d-13be-4b96-b135-0876eeeae452-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.586173 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.585809 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.586173 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.585944 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.586173 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.586036 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f9be1d-13be-4b96-b135-0876eeeae452-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.586173 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.586080 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.586173 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.586118 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.586173 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.586171 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f9be1d-13be-4b96-b135-0876eeeae452-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.586584 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.586198 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vjkk\" (UniqueName: \"kubernetes.io/projected/68f9be1d-13be-4b96-b135-0876eeeae452-kube-api-access-8vjkk\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.586584 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.586438 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.594996 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:00:21.594961 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:00:21.687321 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.687283 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68f9be1d-13be-4b96-b135-0876eeeae452-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.687495 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.687339 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.687495 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.687383 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.687576 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.687517 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f9be1d-13be-4b96-b135-0876eeeae452-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.687612 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.687583 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.687645 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.687628 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.687708 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.687692 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f9be1d-13be-4b96-b135-0876eeeae452-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.687769 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.687721 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vjkk\" (UniqueName: \"kubernetes.io/projected/68f9be1d-13be-4b96-b135-0876eeeae452-kube-api-access-8vjkk\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.687828 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.687792 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.687828 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.687805 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-istio-data\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.687983 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.687911 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-credential-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.688166 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.688144 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-workload-certs\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.688264 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.688244 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f9be1d-13be-4b96-b135-0876eeeae452-istiod-ca-cert\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.688320 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.688257 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-workload-socket\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.689995 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.689970 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-istio-envoy\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.690147 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.690132 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f9be1d-13be-4b96-b135-0876eeeae452-istio-podinfo\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.708795 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.708740 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68f9be1d-13be-4b96-b135-0876eeeae452-istio-token\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.708922 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.708902 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vjkk\" (UniqueName: \"kubernetes.io/projected/68f9be1d-13be-4b96-b135-0876eeeae452-kube-api-access-8vjkk\") pod \"e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.725553 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.725519 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:21.779097 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.779059 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:21.882650 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.881358 5300 generic.go:356] "Generic (PLEG): container finished" podID="e9e65cfb-4f44-417e-b378-f6f87da940f5" containerID="8aa2b82bae8bb11de557df02535b673abb4e7c7f9be3b09b1f92dfe18bf24891" exitCode=60 Oct 30 01:00:21.882650 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.881428 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c" event={"ID":"e9e65cfb-4f44-417e-b378-f6f87da940f5","Type":"ContainerDied","Data":"8aa2b82bae8bb11de557df02535b673abb4e7c7f9be3b09b1f92dfe18bf24891"} Oct 30 01:00:21.882650 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.881482 5300 scope.go:122] "RemoveContainer" containerID="d11d9fb8fbd7d15d5c06da312e136bfde43970e87687634c3bc37da8cccd7dde" Oct 30 01:00:21.882650 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:21.881993 5300 scope.go:122] "RemoveContainer" containerID="8aa2b82bae8bb11de557df02535b673abb4e7c7f9be3b09b1f92dfe18bf24891" Oct 30 01:00:21.882650 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:00:21.882342 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c_odh-ai-gateway-infra(e9e65cfb-4f44-417e-b378-f6f87da940f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c" podUID="e9e65cfb-4f44-417e-b378-f6f87da940f5" Oct 30 01:00:22.116434 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:22.116404 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm"] Oct 30 01:00:22.118719 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:00:22.118693 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f9be1d_13be_4b96_b135_0876eeeae452.slice/crio-513f51888a92a15d62b316df9899ddcc806e5d2521764996d941104f00267e5c WatchSource:0}: Error finding container 513f51888a92a15d62b316df9899ddcc806e5d2521764996d941104f00267e5c: Status 404 returned error can't find the container with id 513f51888a92a15d62b316df9899ddcc806e5d2521764996d941104f00267e5c Oct 30 01:00:22.121332 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:22.121300 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:00:22.121413 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:22.121365 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:00:22.121413 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:22.121397 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:00:22.889335 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:22.889294 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" event={"ID":"68f9be1d-13be-4b96-b135-0876eeeae452","Type":"ContainerStarted","Data":"43fab1c20bdc7183c21ece000292fd113e0e1df39d82db9af757b135ba0cb904"} Oct 30 01:00:22.889813 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:22.889357 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" event={"ID":"68f9be1d-13be-4b96-b135-0876eeeae452","Type":"ContainerStarted","Data":"513f51888a92a15d62b316df9899ddcc806e5d2521764996d941104f00267e5c"} Oct 30 01:00:22.971151 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:22.971098 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" podStartSLOduration=1.971084958 podStartE2EDuration="1.971084958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:22.95550239 +0000 UTC m=+1023.271370725" watchObservedRunningTime="2026-10-30 01:00:22.971084958 +0000 UTC m=+1023.286953282" Oct 30 01:00:22.974641 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:22.974618 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c"] Oct 30 01:00:23.178805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:23.178755 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c" Oct 30 01:00:23.350925 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:23.350893 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e65cfb-4f44-417e-b378-f6f87da940f5-kube-api-access-967xx\" (UniqueName: \"kubernetes.io/projected/e9e65cfb-4f44-417e-b378-f6f87da940f5-kube-api-access-967xx\") pod \"e9e65cfb-4f44-417e-b378-f6f87da940f5\" (UID: \"e9e65cfb-4f44-417e-b378-f6f87da940f5\") " Oct 30 01:00:23.351245 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:23.351221 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9e65cfb-4f44-417e-b378-f6f87da940f5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9e65cfb-4f44-417e-b378-f6f87da940f5-maas-api-service-ca\") pod \"e9e65cfb-4f44-417e-b378-f6f87da940f5\" (UID: \"e9e65cfb-4f44-417e-b378-f6f87da940f5\") " Oct 30 01:00:23.351639 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:23.351607 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e65cfb-4f44-417e-b378-f6f87da940f5-maas-api-service-ca" pod "e9e65cfb-4f44-417e-b378-f6f87da940f5" (UID: "e9e65cfb-4f44-417e-b378-f6f87da940f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:00:23.351799 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:23.351754 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9e65cfb-4f44-417e-b378-f6f87da940f5-maas-api-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:00:23.353208 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:23.353185 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e65cfb-4f44-417e-b378-f6f87da940f5-kube-api-access-967xx" pod "e9e65cfb-4f44-417e-b378-f6f87da940f5" (UID: "e9e65cfb-4f44-417e-b378-f6f87da940f5"). InnerVolumeSpecName "kube-api-access-967xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:00:23.453435 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:23.453342 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-967xx\" (UniqueName: \"kubernetes.io/projected/e9e65cfb-4f44-417e-b378-f6f87da940f5-kube-api-access-967xx\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:00:23.553206 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:23.553171 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:23.780151 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:23.780114 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:23.781870 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:23.781844 5300 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 01:00:23.782042 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:23.781892 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" podUID="68f9be1d-13be-4b96-b135-0876eeeae452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 01:00:23.898521 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:23.898494 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c" Oct 30 01:00:23.898987 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:23.898540 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c" event={"ID":"e9e65cfb-4f44-417e-b378-f6f87da940f5","Type":"ContainerDied","Data":"a9c87456ce8658ce8387e772804e8684ab40f4f01116ff08e5d18a2f13cd3d6b"} Oct 30 01:00:23.898987 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:23.898605 5300 scope.go:122] "RemoveContainer" containerID="8aa2b82bae8bb11de557df02535b673abb4e7c7f9be3b09b1f92dfe18bf24891" Oct 30 01:00:23.963781 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:23.963717 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c"] Oct 30 01:00:23.967521 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:23.967497 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c104b8-29888700-8qv9c"] Oct 30 01:00:24.780820 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:24.779931 5300 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 01:00:24.780820 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:24.779995 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" podUID="68f9be1d-13be-4b96-b135-0876eeeae452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 01:00:25.366307 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:25.366274 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:25.780150 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:25.780119 5300 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 01:00:25.780341 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:25.780170 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" podUID="68f9be1d-13be-4b96-b135-0876eeeae452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 01:00:25.845774 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:25.845722 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9e65cfb-4f44-417e-b378-f6f87da940f5" path="/var/lib/kubelet/pods/e9e65cfb-4f44-417e-b378-f6f87da940f5/volumes" Oct 30 01:00:26.779592 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:26.779552 5300 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 01:00:26.780028 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:26.779615 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" podUID="68f9be1d-13be-4b96-b135-0876eeeae452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 01:00:27.115175 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:27.115058 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:27.779737 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:27.779703 5300 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 01:00:27.780150 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:27.779785 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" podUID="68f9be1d-13be-4b96-b135-0876eeeae452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 01:00:28.780542 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:28.780501 5300 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 01:00:28.780979 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:28.780567 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" podUID="68f9be1d-13be-4b96-b135-0876eeeae452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 01:00:28.893058 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:28.893015 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:29.784142 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:29.784112 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:29.784536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:29.784501 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:29.785282 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:29.785262 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:00:30.195572 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:30.195474 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:30.478731 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:30.478690 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:32.351925 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:32.351892 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:33.906234 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:33.906198 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:35.560133 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:35.560095 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:37.365001 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:37.364963 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:39.033937 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:39.033886 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:39.303843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:39.299916 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:39.549787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:39.540910 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:39.631455 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:39.631343 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:39.633064 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:39.633034 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:39.633560 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:39.633536 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" podUID="d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:39.845787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:39.839052 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 01:00:39.872973 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:39.872840 5300 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:40.035788 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:40.034862 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:40.035788 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:40.035324 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 01:00:40.165786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:40.164870 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" podUID="1166fd98-968e-4fda-a0b9-0cfea1ed9e03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:40.337870 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:40.335685 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:40.337870 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:40.336547 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 01:00:40.657843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:40.543207 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:40.657843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:40.544948 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 01:00:40.657843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:40.643402 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" podUID="d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:40.657843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:40.653878 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:40.683468 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:40.672750 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 30 01:00:40.683468 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:40.673173 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 01:00:40.683468 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:40.673259 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 01:00:40.683468 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:40.673314 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 01:00:40.917846 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:40.843877 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:40.917846 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:40.844247 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 01:00:41.038895 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:41.038851 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:41.177046 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:41.176947 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" podUID="1166fd98-968e-4fda-a0b9-0cfea1ed9e03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:41.246480 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:41.246442 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 01:00:41.360882 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:41.360832 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:41.549034 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:41.548990 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:41.667112 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:41.658849 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" podUID="d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 01:00:41.683889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:41.675020 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:41.704810 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:41.684159 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 01:00:41.870228 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:41.867906 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 01:00:41.963001 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:41.962960 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 01:00:42.033664 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:42.031407 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 01:00:42.065570 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:42.065528 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 01:00:42.086192 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:42.086158 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:42.092613 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:42.092567 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 01:00:42.177975 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:42.177862 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" podUID="1166fd98-968e-4fda-a0b9-0cfea1ed9e03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:42.263079 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:42.262991 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 01:00:42.309106 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:42.309071 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 01:00:42.362020 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:42.361971 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:00:42.413230 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:42.407914 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 01:00:42.413230 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:42.407967 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 01:00:42.413230 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:42.410243 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 01:00:43.970611 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:43.970564 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:45.727349 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:45.727316 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:47.548485 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:47.548446 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:49.126842 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:49.126801 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:50.732184 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:50.732146 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:51.600198 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:00:51.600161 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:00:52.569427 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:52.569385 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:54.141531 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:54.141499 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:55.958161 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:55.958130 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:57.525889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:57.525828 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:58.822497 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:58.822449 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm"] Oct 30 01:00:58.823251 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:58.823229 5300 cpu_manager.go:414] "containerMap: removing container" podUID="e9e65cfb-4f44-417e-b378-f6f87da940f5" containerName="cleanup" Oct 30 01:00:58.823370 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:58.823252 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e65cfb-4f44-417e-b378-f6f87da940f5" containerName="cleanup" Oct 30 01:00:58.823370 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:58.823280 5300 cpu_manager.go:414] "containerMap: removing container" podUID="e9e65cfb-4f44-417e-b378-f6f87da940f5" containerName="cleanup" Oct 30 01:00:58.823370 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:58.823289 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e65cfb-4f44-417e-b378-f6f87da940f5" containerName="cleanup" Oct 30 01:00:58.823524 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:58.823461 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e65cfb-4f44-417e-b378-f6f87da940f5" containerName="cleanup" Oct 30 01:00:58.823524 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:58.823476 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e65cfb-4f44-417e-b378-f6f87da940f5" containerName="cleanup" Oct 30 01:00:58.823524 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:58.823490 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e65cfb-4f44-417e-b378-f6f87da940f5" containerName="cleanup" Oct 30 01:00:58.828592 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:58.828571 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" Oct 30 01:00:58.850815 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:58.850789 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm"] Oct 30 01:00:58.861416 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:58.861302 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 01:00:58.861416 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:58.861336 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 01:00:58.861416 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:58.861335 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-txv2f\"" Oct 30 01:00:58.861416 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:58.861390 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 01:00:58.933047 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:58.933013 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d744ad6-75be-4a86-be0f-946122f5302e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v5jpm\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" Oct 30 01:00:58.933213 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:58.933065 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7d744ad6-75be-4a86-be0f-946122f5302e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v5jpm\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" Oct 30 01:00:58.933213 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:58.933195 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7d744ad6-75be-4a86-be0f-946122f5302e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v5jpm\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" Oct 30 01:00:58.933293 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:58.933256 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7d744ad6-75be-4a86-be0f-946122f5302e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v5jpm\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" Oct 30 01:00:58.933415 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:58.933389 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d744ad6-75be-4a86-be0f-946122f5302e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v5jpm\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" Oct 30 01:00:58.933451 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:58.933430 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nprln\" (UniqueName: \"kubernetes.io/projected/7d744ad6-75be-4a86-be0f-946122f5302e-kube-api-access-nprln\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v5jpm\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" Oct 30 01:00:58.933613 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:58.933594 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7d744ad6-75be-4a86-be0f-946122f5302e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v5jpm\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" Oct 30 01:00:59.034884 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:59.034846 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7d744ad6-75be-4a86-be0f-946122f5302e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v5jpm\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" Oct 30 01:00:59.035058 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:59.034949 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d744ad6-75be-4a86-be0f-946122f5302e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v5jpm\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" Oct 30 01:00:59.035058 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:59.034981 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7d744ad6-75be-4a86-be0f-946122f5302e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v5jpm\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" Oct 30 01:00:59.035058 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:59.035018 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7d744ad6-75be-4a86-be0f-946122f5302e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v5jpm\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" Oct 30 01:00:59.035058 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:59.035054 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7d744ad6-75be-4a86-be0f-946122f5302e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v5jpm\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" Oct 30 01:00:59.035277 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:59.035118 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d744ad6-75be-4a86-be0f-946122f5302e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v5jpm\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" Oct 30 01:00:59.035277 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:59.035144 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nprln\" (UniqueName: \"kubernetes.io/projected/7d744ad6-75be-4a86-be0f-946122f5302e-kube-api-access-nprln\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v5jpm\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" Oct 30 01:00:59.035794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:59.035752 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7d744ad6-75be-4a86-be0f-946122f5302e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v5jpm\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" Oct 30 01:00:59.037393 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:59.037368 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7d744ad6-75be-4a86-be0f-946122f5302e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v5jpm\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" Oct 30 01:00:59.037500 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:59.037414 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d744ad6-75be-4a86-be0f-946122f5302e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v5jpm\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" Oct 30 01:00:59.037812 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:59.037791 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7d744ad6-75be-4a86-be0f-946122f5302e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v5jpm\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" Oct 30 01:00:59.037861 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:59.037834 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7d744ad6-75be-4a86-be0f-946122f5302e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v5jpm\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" Oct 30 01:00:59.056181 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:59.056152 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d744ad6-75be-4a86-be0f-946122f5302e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v5jpm\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" Oct 30 01:00:59.056277 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:59.056210 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nprln\" (UniqueName: \"kubernetes.io/projected/7d744ad6-75be-4a86-be0f-946122f5302e-kube-api-access-nprln\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v5jpm\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" Oct 30 01:00:59.140996 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:59.140912 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" Oct 30 01:00:59.259670 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:59.259640 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:00:59.457137 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:59.457108 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm"] Oct 30 01:00:59.457556 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:00:59.457530 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d744ad6_75be_4a86_be0f_946122f5302e.slice/crio-493fa66a5fb35e6b746caae690849203870efb7d23f16c5dbd4232e03edb6ec4 WatchSource:0}: Error finding container 493fa66a5fb35e6b746caae690849203870efb7d23f16c5dbd4232e03edb6ec4: Status 404 returned error can't find the container with id 493fa66a5fb35e6b746caae690849203870efb7d23f16c5dbd4232e03edb6ec4 Oct 30 01:00:59.459424 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:00:59.459407 5300 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:01:00.225451 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:00.225402 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" event={"ID":"7d744ad6-75be-4a86-be0f-946122f5302e","Type":"ContainerStarted","Data":"493fa66a5fb35e6b746caae690849203870efb7d23f16c5dbd4232e03edb6ec4"} Oct 30 01:01:00.958215 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:00.958179 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:02.011137 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:02.011097 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:01:02.011486 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:02.011165 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:01:02.242681 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:02.242635 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" event={"ID":"7d744ad6-75be-4a86-be0f-946122f5302e","Type":"ContainerStarted","Data":"114e26b76f49c2a234ba1546916925d3791474ba8cc6e90d484586ee10da5b9f"} Oct 30 01:01:02.242871 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:02.242700 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" Oct 30 01:01:02.289284 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:02.289193 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" podStartSLOduration=1.738853084 podStartE2EDuration="4.289179815s" totalImagesPullingTime="2.550326731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:02.285869777 +0000 UTC m=+1062.601738103" watchObservedRunningTime="2026-10-30 01:01:02.289179815 +0000 UTC m=+1062.605048139" Oct 30 01:01:02.416692 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:02.416656 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:02.502007 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:02.501964 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:02.853240 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:02.853200 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:03.253277 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:03.253002 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" Oct 30 01:01:04.984994 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:04.984933 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 30 01:01:05.023065 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:05.021870 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:05.097050 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:05.095941 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 01:01:05.187952 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:05.187895 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" podUID="1166fd98-968e-4fda-a0b9-0cfea1ed9e03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:05.364981 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:05.364865 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:05.411907 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:05.411852 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 01:01:05.413324 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:05.412832 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" podUID="d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 01:01:05.585786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:05.549872 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:05.796334 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:05.795932 5300 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" podUID="d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:06.009787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:05.986918 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 30 01:01:06.072463 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:06.055487 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:06.105021 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:06.104975 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:06.175884 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:06.175064 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:06.190390 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:06.188975 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" podUID="1166fd98-968e-4fda-a0b9-0cfea1ed9e03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:06.449603 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:06.449495 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:06.474822 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:06.474711 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:06.548163 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:06.547951 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 01:01:06.584393 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:06.549154 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 01:01:06.585628 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:06.585596 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 01:01:06.585928 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:06.585908 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 01:01:06.589895 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:06.589854 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 01:01:06.705845 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:06.630099 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 01:01:06.705845 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:06.562614 5300 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:06.705845 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:06.589306 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" podUID="d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:06.804415 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:06.804372 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 01:01:06.864786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:06.841553 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 01:01:06.888498 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:06.888330 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 01:01:07.168978 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:07.168920 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 01:01:07.321008 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:07.312964 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" podUID="1166fd98-968e-4fda-a0b9-0cfea1ed9e03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 01:01:07.321341 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:07.313064 5300 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded" Oct 30 01:01:07.321506 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:07.313087 5300 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" podUID="1166fd98-968e-4fda-a0b9-0cfea1ed9e03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": context deadline exceeded" Oct 30 01:01:07.321632 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:07.320684 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn"] Oct 30 01:01:07.335876 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:07.335838 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:07.359738 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:07.336984 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 01:01:07.412792 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:07.399327 5300 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:07.449843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:07.446193 5300 cpu_manager.go:414] "containerMap: removing container" podUID="e9e65cfb-4f44-417e-b378-f6f87da940f5" containerName="cleanup" Oct 30 01:01:07.449843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:07.446225 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e65cfb-4f44-417e-b378-f6f87da940f5" containerName="cleanup" Oct 30 01:01:07.451113 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:07.451076 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 01:01:07.484613 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:07.484570 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:07.788167 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:07.784907 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" podUID="d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 01:01:07.922593 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:07.922530 5300 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:07.922791 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:07.922653 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 01:01:08.347779 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.334630 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 01:01:08.377962 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.375795 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:08.390305 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.390277 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.462847 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.462815 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn"] Oct 30 01:01:08.467865 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.463180 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 01:01:08.468099 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.468084 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 01:01:08.468213 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.468205 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 01:01:08.470447 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.470432 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 01:01:08.470613 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.470602 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 01:01:08.470720 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.470712 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 01:01:08.471014 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.471001 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 01:01:08.477202 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.477181 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 01:01:08.594176 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.594140 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmlll\" (UniqueName: \"kubernetes.io/projected/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-kube-api-access-tmlll\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.594365 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.594334 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.594445 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.594427 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.594506 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.594470 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.594654 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.594623 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.594873 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.594849 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.595016 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.594992 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.595136 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.595118 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.595226 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.595207 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.696667 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.696582 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.696667 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.696647 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.696913 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.696691 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.696913 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.696753 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmlll\" (UniqueName: \"kubernetes.io/projected/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-kube-api-access-tmlll\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.696913 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.696854 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.696913 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.696896 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.697129 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.696926 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.697129 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.696985 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.697129 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.697121 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.697488 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.697441 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-workload-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.698036 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.697973 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-workload-certs\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.698155 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.698122 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istiod-ca-cert\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.698229 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.698207 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-data\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.698407 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.698358 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-credential-socket\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.700076 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.699344 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-envoy\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.700751 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.700729 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-podinfo\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.718566 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.718531 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmlll\" (UniqueName: \"kubernetes.io/projected/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-kube-api-access-tmlll\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.718686 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.718639 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-token\") pod \"e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:08.776980 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:08.776947 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:09.099232 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:09.099205 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn"] Oct 30 01:01:09.099870 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:01:09.099846 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22aacbb7_595e_4895_bcd4_5fec9c9d78e9.slice/crio-4b6dd67d891c5eac6e7d1d4ab6bc9bdc3abfab7cd02e3adc9a751968646bbf96 WatchSource:0}: Error finding container 4b6dd67d891c5eac6e7d1d4ab6bc9bdc3abfab7cd02e3adc9a751968646bbf96: Status 404 returned error can't find the container with id 4b6dd67d891c5eac6e7d1d4ab6bc9bdc3abfab7cd02e3adc9a751968646bbf96 Oct 30 01:01:09.102508 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:09.102468 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:01:09.102577 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:09.102555 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:01:09.102620 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:09.102605 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:01:09.215333 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:09.215300 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:09.709526 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:09.709486 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" event={"ID":"22aacbb7-595e-4895-bcd4-5fec9c9d78e9","Type":"ContainerStarted","Data":"a6ef06136594319e95049c1f8719e92eb131384c1ba6f735997606f88a6195c4"} Oct 30 01:01:09.709993 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:09.709604 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" event={"ID":"22aacbb7-595e-4895-bcd4-5fec9c9d78e9","Type":"ContainerStarted","Data":"4b6dd67d891c5eac6e7d1d4ab6bc9bdc3abfab7cd02e3adc9a751968646bbf96"} Oct 30 01:01:09.753984 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:09.753919 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" podStartSLOduration=3.753905268 podStartE2EDuration="3.753905268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:09.753379695 +0000 UTC m=+1070.069248021" watchObservedRunningTime="2026-10-30 01:01:09.753905268 +0000 UTC m=+1070.069773592" Oct 30 01:01:09.777052 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:09.777022 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:10.779211 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:10.779173 5300 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:01:10.779624 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:10.779230 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" podUID="22aacbb7-595e-4895-bcd4-5fec9c9d78e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 01:01:10.938252 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:10.938213 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:11.777483 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:11.777448 5300 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:01:11.777649 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:11.777501 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" podUID="22aacbb7-595e-4895-bcd4-5fec9c9d78e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 01:01:12.642495 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:12.642422 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:12.777920 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:12.777888 5300 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:01:12.778126 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:12.777941 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" podUID="22aacbb7-595e-4895-bcd4-5fec9c9d78e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 01:01:13.777637 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:13.777602 5300 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:01:13.778077 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:13.777667 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" podUID="22aacbb7-595e-4895-bcd4-5fec9c9d78e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 01:01:14.472038 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:14.471996 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:14.778267 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:14.778177 5300 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:01:14.778267 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:14.778239 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" podUID="22aacbb7-595e-4895-bcd4-5fec9c9d78e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 01:01:15.777808 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:15.777751 5300 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:01:15.778025 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:15.777837 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" podUID="22aacbb7-595e-4895-bcd4-5fec9c9d78e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 01:01:16.154210 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:16.154111 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:16.932656 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:16.932622 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:16.933434 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:16.933367 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:16.934448 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:16.934423 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:01:17.826399 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:17.826363 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:19.585950 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:19.585907 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:21.148804 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:21.148747 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:21.595005 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:01:21.594969 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:01:23.229201 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:23.229157 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:24.847278 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:24.847242 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:26.548728 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:26.548692 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:28.442567 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:28.442529 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:29.405699 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:29.405655 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:29.506466 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:29.506429 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:30.176435 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:30.176396 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:30.555928 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:30.551115 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:31.340787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:31.339873 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:31.344787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:31.341189 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" podUID="1166fd98-968e-4fda-a0b9-0cfea1ed9e03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 01:01:31.396788 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:31.395877 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:31.416788 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:31.397128 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:31.485788 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:31.484860 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:31.507787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:31.499856 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 01:01:31.791819 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:31.791734 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" podUID="d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:31.943859 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:31.943812 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:32.342474 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:32.342417 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" podUID="1166fd98-968e-4fda-a0b9-0cfea1ed9e03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:32.343640 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:32.343616 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 01:01:32.344137 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:32.344094 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 01:01:32.346373 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:32.345456 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 01:01:32.407816 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:32.402019 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:32.407816 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:32.402684 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 01:01:32.615061 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:32.614958 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:32.640869 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:32.615893 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 01:01:32.659106 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:32.659054 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:32.659741 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:32.659718 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 01:01:32.815918 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:32.815859 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" podUID="d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:32.841787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:32.816845 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 01:01:32.978786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:32.973956 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:32.999792 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:32.996955 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 01:01:33.275960 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:33.275858 5300 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:01:33.298250 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:33.298204 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" podUID="22aacbb7-595e-4895-bcd4-5fec9c9d78e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:33.366787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:33.345956 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:33.378817 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:33.352196 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" podUID="1166fd98-968e-4fda-a0b9-0cfea1ed9e03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 01:01:33.387625 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:33.387583 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:33.405946 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:33.405877 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 01:01:33.778788 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:33.760183 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 01:01:33.786395 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:33.786356 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:33.890786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:33.873108 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" podUID="d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:34.010745 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:34.010696 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:34.377111 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:34.375094 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:34.393095 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:34.393049 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:34.418933 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:34.418884 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:34.419707 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:34.397009 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" podUID="1166fd98-968e-4fda-a0b9-0cfea1ed9e03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:34.566377 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:34.540140 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 01:01:34.823786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:34.796596 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:34.850792 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:34.802897 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:34.919988 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:34.918011 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" podUID="d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:35.002340 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:35.002311 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 01:01:35.025977 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:35.025922 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:35.056190 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:35.056157 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 01:01:35.310752 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:35.310722 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 01:01:35.474787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:35.457789 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 01:01:35.474787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:35.457980 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 01:01:35.474787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:35.467151 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 01:01:35.474787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:35.471791 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 01:01:35.474787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:35.473415 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 01:01:35.497447 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:35.497419 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 01:01:35.501085 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:35.501013 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:37.172165 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:37.172131 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:38.838348 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:38.838311 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:40.606229 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:40.606188 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:42.204215 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:42.204177 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:43.938972 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:43.938928 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:45.638102 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:45.638070 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:45.742314 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:45.742270 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:47.350143 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:47.350108 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:47.578663 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:47.578630 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:48.058226 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:48.058185 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:48.210358 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:48.210323 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:48.571969 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:48.571918 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:48.669127 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:48.669086 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:49.177030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:49.176996 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:49.271211 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:49.271174 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:51.156500 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:51.156472 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:51.273323 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:51.273285 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:51.386474 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:51.386424 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:51.594356 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:01:51.594316 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:01:51.707176 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:51.707147 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:52.540775 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:52.540733 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:52.774856 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:52.774748 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:53.027103 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:53.027056 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:53.124352 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:53.124322 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:54.855167 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:54.855137 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:55.371349 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:55.371313 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:55.474845 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:55.474815 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:55.575824 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:55.575786 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:55.675463 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:55.675382 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:55.883722 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:55.883693 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:56.457071 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:56.457028 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:56.557607 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:56.557570 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:56.730175 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:56.730130 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:57.849076 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:57.849041 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:58.006271 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:58.005798 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:01:58.504785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:58.435617 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" podUID="1166fd98-968e-4fda-a0b9-0cfea1ed9e03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 01:01:58.551836 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:58.447723 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:58.639785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:58.502098 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 01:01:58.639785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:58.507607 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:58.789841 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:58.786144 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 01:01:58.872044 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:58.871992 5300 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:58.899947 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:58.899900 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" podUID="d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:59.018786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:59.015567 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:59.091091 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:59.089856 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:59.607786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:59.576750 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" podUID="1166fd98-968e-4fda-a0b9-0cfea1ed9e03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:59.607786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:59.598879 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:59.609232 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:59.609205 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 01:01:59.609453 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:59.609437 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 01:01:59.618997 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:59.618960 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:59.628806 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:59.623154 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 01:01:59.683226 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:59.626322 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 01:01:59.683560 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:59.683538 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 01:01:59.711786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:59.686264 5300 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:01:59.712102 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:59.712047 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" podUID="e7d2e040-342a-4ea4-a23d-c36722f2d64e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:59.805063 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:59.805017 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:59.830048 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:59.830006 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 01:01:59.874836 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:59.867245 5300 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:01:59.923598 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:01:59.923549 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" podUID="d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:00.056501 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:00.056457 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:00.241873 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:00.063970 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 01:02:00.241873 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:00.143013 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 01:02:00.243931 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:00.238917 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 01:02:00.244435 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:00.244419 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 01:02:00.632842 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:00.631681 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 01:02:00.651053 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:00.650834 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 01:02:00.844970 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:00.651374 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" podUID="1166fd98-968e-4fda-a0b9-0cfea1ed9e03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 01:02:00.844970 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:00.683721 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 01:02:00.844970 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:00.775865 5300 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:02:00.844970 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:00.775958 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" podUID="59536065-cfa0-4f0c-a2ca-55284c369d81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:00.860588 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:00.845400 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 01:02:00.879089 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:00.878868 5300 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:02:00.879089 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:00.878947 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" podUID="68f9be1d-13be-4b96-b135-0876eeeae452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:00.908751 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:00.898747 5300 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:02:00.908751 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:00.898829 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" podUID="ba5bee00-9a0a-4a43-abea-35da290cbcaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:00.942785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:00.912081 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:00.957785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:00.943089 5300 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:02:00.957785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:00.943151 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" podUID="12e75f96-d218-400a-8b56-0cea344f3e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:00.957785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:00.944802 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" podUID="d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 01:02:01.266928 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:01.266880 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 30 01:02:01.286580 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:01.286516 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:01.702300 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:01.699862 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" podUID="1166fd98-968e-4fda-a0b9-0cfea1ed9e03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 01:02:01.702300 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:01.700210 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 01:02:01.777188 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:01.777125 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 01:02:01.785660 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:01.784334 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 01:02:01.808281 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:01.808164 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" Oct 30 01:02:01.850332 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:01.850282 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:01.889778 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:01.889735 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 01:02:01.890271 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:01.889842 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 01:02:01.892873 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:01.892852 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 01:02:01.922306 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:01.916486 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 01:02:01.922306 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:01.917618 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 01:02:01.945230 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:01.945130 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 01:02:01.948947 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:01.948060 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 01:02:02.693057 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:02.693023 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:02.789430 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:02.789392 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:03.094856 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:03.094819 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:03.196283 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:03.196253 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:03.329706 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:03.329670 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:04.984906 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:04.984855 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:05.080076 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:05.080038 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:05.162037 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:05.161995 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts"] Oct 30 01:02:05.166273 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:05.166248 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts" Oct 30 01:02:05.183962 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:05.183939 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts"] Oct 30 01:02:05.194588 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:05.194564 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b61ed672\"" Oct 30 01:02:05.194690 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:05.194577 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b61ed672-dockercfg-6fb6v\"" Oct 30 01:02:05.258462 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:05.258376 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:05.264272 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:05.264217 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81c86f2-0133-4497-b427-798427782cb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts\" (UID: \"e81c86f2-0133-4497-b427-798427782cb6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts" Oct 30 01:02:05.264617 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:05.264596 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shv8h\" (UniqueName: \"kubernetes.io/projected/e81c86f2-0133-4497-b427-798427782cb6-kube-api-access-shv8h\") pod \"payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts\" (UID: \"e81c86f2-0133-4497-b427-798427782cb6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts" Oct 30 01:02:05.366624 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:05.366589 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81c86f2-0133-4497-b427-798427782cb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts\" (UID: \"e81c86f2-0133-4497-b427-798427782cb6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts" Oct 30 01:02:05.366831 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:05.366728 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shv8h\" (UniqueName: \"kubernetes.io/projected/e81c86f2-0133-4497-b427-798427782cb6-kube-api-access-shv8h\") pod \"payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts\" (UID: \"e81c86f2-0133-4497-b427-798427782cb6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts" Oct 30 01:02:05.367399 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:05.367375 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81c86f2-0133-4497-b427-798427782cb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts\" (UID: \"e81c86f2-0133-4497-b427-798427782cb6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts" Oct 30 01:02:05.388161 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:05.388130 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shv8h\" (UniqueName: \"kubernetes.io/projected/e81c86f2-0133-4497-b427-798427782cb6-kube-api-access-shv8h\") pod \"payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts\" (UID: \"e81c86f2-0133-4497-b427-798427782cb6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts" Oct 30 01:02:05.480253 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:05.480216 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts" Oct 30 01:02:05.820146 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:02:05.820113 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode81c86f2_0133_4497_b427_798427782cb6.slice/crio-e8622f672b010f36ff2e54bff1b9c207b128556526ce24965b199fcf1e31af02 WatchSource:0}: Error finding container e8622f672b010f36ff2e54bff1b9c207b128556526ce24965b199fcf1e31af02: Status 404 returned error can't find the container with id e8622f672b010f36ff2e54bff1b9c207b128556526ce24965b199fcf1e31af02 Oct 30 01:02:05.823943 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:05.823917 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts"] Oct 30 01:02:05.836377 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:05.836341 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts" event={"ID":"e81c86f2-0133-4497-b427-798427782cb6","Type":"ContainerStarted","Data":"e8622f672b010f36ff2e54bff1b9c207b128556526ce24965b199fcf1e31af02"} Oct 30 01:02:06.844272 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:06.844231 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts" event={"ID":"e81c86f2-0133-4497-b427-798427782cb6","Type":"ContainerStarted","Data":"3023dc2deb977692ad3389b0ca2a0991bc2973afce6285c3bea3d4bdacf01c02"} Oct 30 01:02:06.844747 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:06.844300 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts" Oct 30 01:02:06.887473 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:06.887419 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts" podStartSLOduration=1.887405432 podStartE2EDuration="1.887405432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:06.885152148 +0000 UTC m=+1127.201020474" watchObservedRunningTime="2026-10-30 01:02:06.887405432 +0000 UTC m=+1127.203273814" Oct 30 01:02:06.950011 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:06.949969 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:08.540330 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:08.540289 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:08.621843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:08.621811 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:08.739103 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:08.739062 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:10.466652 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:10.466618 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:12.156121 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:12.156084 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:13.478094 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:13.478054 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:13.903012 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:13.902925 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:13.984282 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:13.984247 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:14.084640 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:14.084605 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:15.678838 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:15.678794 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:16.338319 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:16.338282 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:17.853548 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:17.853518 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts" Oct 30 01:02:18.597460 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:18.597425 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:19.200380 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:19.200340 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:19.590413 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:19.590376 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:20.736634 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:20.736604 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:20.816529 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:20.816463 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:21.195450 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:21.195367 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:21.610043 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:02:21.610002 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:02:22.534466 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:22.534434 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:24.355274 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:24.355240 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:25.501144 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:25.501108 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:26.187061 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:26.186997 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:27.223940 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:27.223877 5300 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" podUID="1166fd98-968e-4fda-a0b9-0cfea1ed9e03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:27.235913 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:27.235852 5300 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:27.656031 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:27.655920 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:02:28.341182 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:28.341116 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:30.064958 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:30.064925 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:30.998076 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:30.998042 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:31.948442 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:31.948400 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:33.739687 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:33.739650 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:35.458412 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:35.458371 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:37.362030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:37.361988 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:39.021461 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:39.021428 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:40.748988 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:40.748953 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:42.693992 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:42.693956 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:43.348048 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:43.348009 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:43.447077 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:43.447047 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:43.547292 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:43.547261 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:44.510983 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:44.510950 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:46.200244 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:46.200205 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:47.737507 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:47.737464 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:48.226365 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:48.226335 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:48.323328 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:48.323279 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:48.433478 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:48.433442 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:48.538158 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:48.538072 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:49.258397 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:49.258361 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:49.371318 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:49.371282 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:49.505524 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:49.505490 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:49.967121 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:49.967081 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:50.141311 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:50.141275 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:50.243991 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:50.243957 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:50.369288 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:50.369259 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:51.473883 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:51.473806 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:51.568461 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:51.568429 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:51.606671 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:02:51.606635 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:02:51.868886 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:51.864327 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:52.491336 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:52.491301 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:53.300413 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:53.300372 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:53.414333 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:53.414301 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:53.629602 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:53.629515 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:53.730475 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:53.730426 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:54.545867 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:54.545836 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:54.654812 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:54.651329 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:54.749939 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:54.749737 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:54.910223 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:54.910139 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:55.181341 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:55.181257 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t"] Oct 30 01:02:55.185664 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:55.185640 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t" Oct 30 01:02:55.205277 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:55.205254 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t"] Oct 30 01:02:55.218075 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:55.218042 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-837d7038\"" Oct 30 01:02:55.232957 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:55.232924 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e774204f-594a-4ec4-8232-f318fcca84e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t\" (UID: \"e774204f-594a-4ec4-8232-f318fcca84e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t" Oct 30 01:02:55.233131 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:55.233118 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snnm7\" (UniqueName: \"kubernetes.io/projected/e774204f-594a-4ec4-8232-f318fcca84e8-kube-api-access-snnm7\") pod \"payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t\" (UID: \"e774204f-594a-4ec4-8232-f318fcca84e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t" Oct 30 01:02:55.235046 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:55.235024 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-837d7038-dockercfg-h85gq\"" Oct 30 01:02:55.310354 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:55.310321 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm"] Oct 30 01:02:55.314588 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:55.314560 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm" Oct 30 01:02:55.335549 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:55.335519 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm"] Oct 30 01:02:55.338350 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:55.336623 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snnm7\" (UniqueName: \"kubernetes.io/projected/e774204f-594a-4ec4-8232-f318fcca84e8-kube-api-access-snnm7\") pod \"payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t\" (UID: \"e774204f-594a-4ec4-8232-f318fcca84e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t" Oct 30 01:02:55.338350 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:55.337141 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e774204f-594a-4ec4-8232-f318fcca84e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t\" (UID: \"e774204f-594a-4ec4-8232-f318fcca84e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t" Oct 30 01:02:55.338350 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:55.338281 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e774204f-594a-4ec4-8232-f318fcca84e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t\" (UID: \"e774204f-594a-4ec4-8232-f318fcca84e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t" Oct 30 01:02:55.374107 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:55.374074 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snnm7\" (UniqueName: \"kubernetes.io/projected/e774204f-594a-4ec4-8232-f318fcca84e8-kube-api-access-snnm7\") pod \"payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t\" (UID: \"e774204f-594a-4ec4-8232-f318fcca84e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t" Oct 30 01:02:55.438848 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:55.438722 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94fcs\" (UniqueName: \"kubernetes.io/projected/a5f84b76-4029-4f4d-aebf-34103ec9ca17-kube-api-access-94fcs\") pod \"payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm\" (UID: \"a5f84b76-4029-4f4d-aebf-34103ec9ca17\") " pod="openshift-ingress/payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm" Oct 30 01:02:55.439226 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:55.439185 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f84b76-4029-4f4d-aebf-34103ec9ca17-plugins-config-volume\") pod \"payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm\" (UID: \"a5f84b76-4029-4f4d-aebf-34103ec9ca17\") " pod="openshift-ingress/payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm" Oct 30 01:02:55.499812 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:55.499743 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t" Oct 30 01:02:55.540840 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:55.540796 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94fcs\" (UniqueName: \"kubernetes.io/projected/a5f84b76-4029-4f4d-aebf-34103ec9ca17-kube-api-access-94fcs\") pod \"payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm\" (UID: \"a5f84b76-4029-4f4d-aebf-34103ec9ca17\") " pod="openshift-ingress/payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm" Oct 30 01:02:55.541131 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:55.541100 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f84b76-4029-4f4d-aebf-34103ec9ca17-plugins-config-volume\") pod \"payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm\" (UID: \"a5f84b76-4029-4f4d-aebf-34103ec9ca17\") " pod="openshift-ingress/payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm" Oct 30 01:02:55.541972 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:55.541943 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f84b76-4029-4f4d-aebf-34103ec9ca17-plugins-config-volume\") pod \"payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm\" (UID: \"a5f84b76-4029-4f4d-aebf-34103ec9ca17\") " pod="openshift-ingress/payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm" Oct 30 01:02:55.561623 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:55.561573 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94fcs\" (UniqueName: \"kubernetes.io/projected/a5f84b76-4029-4f4d-aebf-34103ec9ca17-kube-api-access-94fcs\") pod \"payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm\" (UID: \"a5f84b76-4029-4f4d-aebf-34103ec9ca17\") " pod="openshift-ingress/payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm" Oct 30 01:02:55.629012 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:55.628932 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm" Oct 30 01:02:55.732529 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:55.729671 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts"] Oct 30 01:02:55.848731 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:55.848190 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t"] Oct 30 01:02:55.849186 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:55.848906 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:55.954729 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:55.954696 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm"] Oct 30 01:02:55.955252 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:02:55.955223 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f84b76_4029_4f4d_aebf_34103ec9ca17.slice/crio-9957b28239d51f997b1f5f80d1df8305dafd58f02a58a6f9f497c54a8734cee5 WatchSource:0}: Error finding container 9957b28239d51f997b1f5f80d1df8305dafd58f02a58a6f9f497c54a8734cee5: Status 404 returned error can't find the container with id 9957b28239d51f997b1f5f80d1df8305dafd58f02a58a6f9f497c54a8734cee5 Oct 30 01:02:55.961729 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:55.960965 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:56.389270 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:56.389232 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm" event={"ID":"a5f84b76-4029-4f4d-aebf-34103ec9ca17","Type":"ContainerStarted","Data":"1e113a42d2dfcf986fd0729b1fbcd06df362bb879d1b833c9737495086637a6a"} Oct 30 01:02:56.389459 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:56.389296 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm" event={"ID":"a5f84b76-4029-4f4d-aebf-34103ec9ca17","Type":"ContainerStarted","Data":"9957b28239d51f997b1f5f80d1df8305dafd58f02a58a6f9f497c54a8734cee5"} Oct 30 01:02:56.389459 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:56.389327 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm" Oct 30 01:02:56.390971 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:56.390946 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t" event={"ID":"e774204f-594a-4ec4-8232-f318fcca84e8","Type":"ContainerStarted","Data":"47d9cb75ff89b477399eff78287d48e4c07caabc94d8896aeebad4f8ee7e1b9e"} Oct 30 01:02:56.391162 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:56.390989 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t" event={"ID":"e774204f-594a-4ec4-8232-f318fcca84e8","Type":"ContainerStarted","Data":"6f9276f9c378605c4d85e2b53a89bfe1b809a356bbb6237336d218a5adf5def2"} Oct 30 01:02:56.391162 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:56.391017 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t" Oct 30 01:02:56.509642 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:56.509454 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t" podStartSLOduration=1.509430141 podStartE2EDuration="1.509430141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:56.506848248 +0000 UTC m=+1176.822716575" watchObservedRunningTime="2026-10-30 01:02:56.509430141 +0000 UTC m=+1176.825298497" Oct 30 01:02:56.511783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:56.511720 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm" podStartSLOduration=1.511702176 podStartE2EDuration="1.511702176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:56.461622633 +0000 UTC m=+1176.777490959" watchObservedRunningTime="2026-10-30 01:02:56.511702176 +0000 UTC m=+1176.827570503" Oct 30 01:02:56.545792 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:56.545727 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:57.975216 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:57.975181 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:58.530235 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:58.530193 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:58.637200 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:58.637169 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:02:58.738959 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:02:58.738921 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:00.184627 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:00.184599 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:00.458442 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:00.458352 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:00.730888 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:00.730783 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts" podUID="e81c86f2-0133-4497-b427-798427782cb6" containerName="payload-pre-processing" containerID="cri-o://3023dc2deb977692ad3389b0ca2a0991bc2973afce6285c3bea3d4bdacf01c02" gracePeriod=25 Oct 30 01:03:01.015718 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:01.015688 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts" Oct 30 01:03:01.120968 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:01.120927 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:01.127355 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:01.127319 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e81c86f2-0133-4497-b427-798427782cb6-kube-api-access-shv8h\" (UniqueName: \"kubernetes.io/projected/e81c86f2-0133-4497-b427-798427782cb6-kube-api-access-shv8h\") pod \"e81c86f2-0133-4497-b427-798427782cb6\" (UID: \"e81c86f2-0133-4497-b427-798427782cb6\") " Oct 30 01:03:01.127585 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:01.127565 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e81c86f2-0133-4497-b427-798427782cb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81c86f2-0133-4497-b427-798427782cb6-plugins-config-volume\") pod \"e81c86f2-0133-4497-b427-798427782cb6\" (UID: \"e81c86f2-0133-4497-b427-798427782cb6\") " Oct 30 01:03:01.128075 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:01.128041 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e81c86f2-0133-4497-b427-798427782cb6-plugins-config-volume" pod "e81c86f2-0133-4497-b427-798427782cb6" (UID: "e81c86f2-0133-4497-b427-798427782cb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:01.128203 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:01.128183 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e81c86f2-0133-4497-b427-798427782cb6-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:03:01.129653 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:01.129630 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e81c86f2-0133-4497-b427-798427782cb6-kube-api-access-shv8h" pod "e81c86f2-0133-4497-b427-798427782cb6" (UID: "e81c86f2-0133-4497-b427-798427782cb6"). InnerVolumeSpecName "kube-api-access-shv8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:01.211627 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:01.211593 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:01.229285 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:01.229254 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shv8h\" (UniqueName: \"kubernetes.io/projected/e81c86f2-0133-4497-b427-798427782cb6-kube-api-access-shv8h\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:03:01.432694 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:01.432604 5300 generic.go:356] "Generic (PLEG): container finished" podID="e81c86f2-0133-4497-b427-798427782cb6" containerID="3023dc2deb977692ad3389b0ca2a0991bc2973afce6285c3bea3d4bdacf01c02" exitCode=0 Oct 30 01:03:01.432694 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:01.432675 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts" Oct 30 01:03:01.432694 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:01.432685 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts" event={"ID":"e81c86f2-0133-4497-b427-798427782cb6","Type":"ContainerDied","Data":"3023dc2deb977692ad3389b0ca2a0991bc2973afce6285c3bea3d4bdacf01c02"} Oct 30 01:03:01.432952 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:01.432771 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts" event={"ID":"e81c86f2-0133-4497-b427-798427782cb6","Type":"ContainerDied","Data":"e8622f672b010f36ff2e54bff1b9c207b128556526ce24965b199fcf1e31af02"} Oct 30 01:03:01.432952 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:01.432799 5300 scope.go:122] "RemoveContainer" containerID="3023dc2deb977692ad3389b0ca2a0991bc2973afce6285c3bea3d4bdacf01c02" Oct 30 01:03:01.443222 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:01.443198 5300 scope.go:122] "RemoveContainer" containerID="3023dc2deb977692ad3389b0ca2a0991bc2973afce6285c3bea3d4bdacf01c02" Oct 30 01:03:01.443548 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:03:01.443523 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3023dc2deb977692ad3389b0ca2a0991bc2973afce6285c3bea3d4bdacf01c02\": container with ID starting with 3023dc2deb977692ad3389b0ca2a0991bc2973afce6285c3bea3d4bdacf01c02 not found: ID does not exist" containerID="3023dc2deb977692ad3389b0ca2a0991bc2973afce6285c3bea3d4bdacf01c02" Oct 30 01:03:01.443648 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:01.443556 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3023dc2deb977692ad3389b0ca2a0991bc2973afce6285c3bea3d4bdacf01c02"} err="failed to get container status \"3023dc2deb977692ad3389b0ca2a0991bc2973afce6285c3bea3d4bdacf01c02\": rpc error: code = NotFound desc = could not find container \"3023dc2deb977692ad3389b0ca2a0991bc2973afce6285c3bea3d4bdacf01c02\": container with ID starting with 3023dc2deb977692ad3389b0ca2a0991bc2973afce6285c3bea3d4bdacf01c02 not found: ID does not exist" Oct 30 01:03:01.489912 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:01.489883 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts"] Oct 30 01:03:01.498651 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:01.498624 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b61ed672-65dd489d55-tfgts"] Oct 30 01:03:01.821976 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:01.821937 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e81c86f2-0133-4497-b427-798427782cb6" path="/var/lib/kubelet/pods/e81c86f2-0133-4497-b427-798427782cb6/volumes" Oct 30 01:03:02.059370 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:02.059331 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:03.735225 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:03.735195 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:05.305964 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:05.305917 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:06.699407 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:06.699370 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:07.014700 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:07.014665 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:07.092467 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:07.092431 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:07.402006 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:07.401895 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm" Oct 30 01:03:07.403670 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:07.403646 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t" Oct 30 01:03:07.760527 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:07.760495 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:08.831219 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:08.831144 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:10.568100 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:10.568059 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:12.314984 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:12.314950 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:12.569207 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:12.569126 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:13.908181 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:13.908144 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:15.597474 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:15.597404 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:17.384548 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:17.384516 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:17.489288 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:17.489220 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:18.964526 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:18.964491 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:20.681639 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:20.681568 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:21.594157 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:03:21.594121 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:03:22.488620 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:22.488587 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:23.481716 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:23.481681 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:24.016425 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:24.016392 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:25.843562 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:25.843516 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:27.522967 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:27.522934 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:29.281198 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:29.281165 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:31.110973 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:31.110936 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:31.194847 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:31.194812 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:32.667024 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:32.666992 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:34.406223 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:34.406189 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:36.206633 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:36.206594 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:37.797629 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:37.797596 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:39.253903 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:39.253865 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm"] Oct 30 01:03:39.287608 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:39.287571 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t"] Oct 30 01:03:39.378331 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:39.378297 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:39.477192 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:39.477153 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:41.050333 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:41.050301 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:42.691368 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:42.691334 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:44.255360 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.255313 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm" podUID="a5f84b76-4029-4f4d-aebf-34103ec9ca17" containerName="payload-processing" containerID="cri-o://1e113a42d2dfcf986fd0729b1fbcd06df362bb879d1b833c9737495086637a6a" gracePeriod=25 Oct 30 01:03:44.287960 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.287923 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t" podUID="e774204f-594a-4ec4-8232-f318fcca84e8" containerName="payload-pre-processing" containerID="cri-o://47d9cb75ff89b477399eff78287d48e4c07caabc94d8896aeebad4f8ee7e1b9e" gracePeriod=25 Oct 30 01:03:44.382637 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.382600 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:44.482483 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.482451 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:44.591581 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.591553 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm" Oct 30 01:03:44.626681 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.626647 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f84b76-4029-4f4d-aebf-34103ec9ca17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f84b76-4029-4f4d-aebf-34103ec9ca17-plugins-config-volume\") pod \"a5f84b76-4029-4f4d-aebf-34103ec9ca17\" (UID: \"a5f84b76-4029-4f4d-aebf-34103ec9ca17\") " Oct 30 01:03:44.626887 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.626741 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f84b76-4029-4f4d-aebf-34103ec9ca17-kube-api-access-94fcs\" (UniqueName: \"kubernetes.io/projected/a5f84b76-4029-4f4d-aebf-34103ec9ca17-kube-api-access-94fcs\") pod \"a5f84b76-4029-4f4d-aebf-34103ec9ca17\" (UID: \"a5f84b76-4029-4f4d-aebf-34103ec9ca17\") " Oct 30 01:03:44.627203 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.627146 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f84b76-4029-4f4d-aebf-34103ec9ca17-plugins-config-volume" pod "a5f84b76-4029-4f4d-aebf-34103ec9ca17" (UID: "a5f84b76-4029-4f4d-aebf-34103ec9ca17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:44.627618 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.627595 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f84b76-4029-4f4d-aebf-34103ec9ca17-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:03:44.629191 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.629169 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f84b76-4029-4f4d-aebf-34103ec9ca17-kube-api-access-94fcs" pod "a5f84b76-4029-4f4d-aebf-34103ec9ca17" (UID: "a5f84b76-4029-4f4d-aebf-34103ec9ca17"). InnerVolumeSpecName "kube-api-access-94fcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:44.660017 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.659985 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t" Oct 30 01:03:44.728461 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.728423 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e774204f-594a-4ec4-8232-f318fcca84e8-kube-api-access-snnm7\" (UniqueName: \"kubernetes.io/projected/e774204f-594a-4ec4-8232-f318fcca84e8-kube-api-access-snnm7\") pod \"e774204f-594a-4ec4-8232-f318fcca84e8\" (UID: \"e774204f-594a-4ec4-8232-f318fcca84e8\") " Oct 30 01:03:44.728633 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.728523 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e774204f-594a-4ec4-8232-f318fcca84e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e774204f-594a-4ec4-8232-f318fcca84e8-plugins-config-volume\") pod \"e774204f-594a-4ec4-8232-f318fcca84e8\" (UID: \"e774204f-594a-4ec4-8232-f318fcca84e8\") " Oct 30 01:03:44.728935 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.728917 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94fcs\" (UniqueName: \"kubernetes.io/projected/a5f84b76-4029-4f4d-aebf-34103ec9ca17-kube-api-access-94fcs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:03:44.729066 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.728958 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e774204f-594a-4ec4-8232-f318fcca84e8-plugins-config-volume" pod "e774204f-594a-4ec4-8232-f318fcca84e8" (UID: "e774204f-594a-4ec4-8232-f318fcca84e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:03:44.730960 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.730932 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e774204f-594a-4ec4-8232-f318fcca84e8-kube-api-access-snnm7" pod "e774204f-594a-4ec4-8232-f318fcca84e8" (UID: "e774204f-594a-4ec4-8232-f318fcca84e8"). InnerVolumeSpecName "kube-api-access-snnm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:03:44.743752 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.743722 5300 generic.go:356] "Generic (PLEG): container finished" podID="e774204f-594a-4ec4-8232-f318fcca84e8" containerID="47d9cb75ff89b477399eff78287d48e4c07caabc94d8896aeebad4f8ee7e1b9e" exitCode=0 Oct 30 01:03:44.743921 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.743798 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t" Oct 30 01:03:44.743921 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.743815 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t" event={"ID":"e774204f-594a-4ec4-8232-f318fcca84e8","Type":"ContainerDied","Data":"47d9cb75ff89b477399eff78287d48e4c07caabc94d8896aeebad4f8ee7e1b9e"} Oct 30 01:03:44.743921 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.743881 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t" event={"ID":"e774204f-594a-4ec4-8232-f318fcca84e8","Type":"ContainerDied","Data":"6f9276f9c378605c4d85e2b53a89bfe1b809a356bbb6237336d218a5adf5def2"} Oct 30 01:03:44.743921 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.743914 5300 scope.go:122] "RemoveContainer" containerID="47d9cb75ff89b477399eff78287d48e4c07caabc94d8896aeebad4f8ee7e1b9e" Oct 30 01:03:44.745878 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.745854 5300 generic.go:356] "Generic (PLEG): container finished" podID="a5f84b76-4029-4f4d-aebf-34103ec9ca17" containerID="1e113a42d2dfcf986fd0729b1fbcd06df362bb879d1b833c9737495086637a6a" exitCode=0 Oct 30 01:03:44.745998 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.745971 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm" Oct 30 01:03:44.746116 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.746078 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm" event={"ID":"a5f84b76-4029-4f4d-aebf-34103ec9ca17","Type":"ContainerDied","Data":"1e113a42d2dfcf986fd0729b1fbcd06df362bb879d1b833c9737495086637a6a"} Oct 30 01:03:44.746235 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.746139 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm" event={"ID":"a5f84b76-4029-4f4d-aebf-34103ec9ca17","Type":"ContainerDied","Data":"9957b28239d51f997b1f5f80d1df8305dafd58f02a58a6f9f497c54a8734cee5"} Oct 30 01:03:44.755279 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.755218 5300 scope.go:122] "RemoveContainer" containerID="47d9cb75ff89b477399eff78287d48e4c07caabc94d8896aeebad4f8ee7e1b9e" Oct 30 01:03:44.755566 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:03:44.755544 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d9cb75ff89b477399eff78287d48e4c07caabc94d8896aeebad4f8ee7e1b9e\": container with ID starting with 47d9cb75ff89b477399eff78287d48e4c07caabc94d8896aeebad4f8ee7e1b9e not found: ID does not exist" containerID="47d9cb75ff89b477399eff78287d48e4c07caabc94d8896aeebad4f8ee7e1b9e" Oct 30 01:03:44.755657 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.755577 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d9cb75ff89b477399eff78287d48e4c07caabc94d8896aeebad4f8ee7e1b9e"} err="failed to get container status \"47d9cb75ff89b477399eff78287d48e4c07caabc94d8896aeebad4f8ee7e1b9e\": rpc error: code = NotFound desc = could not find container \"47d9cb75ff89b477399eff78287d48e4c07caabc94d8896aeebad4f8ee7e1b9e\": container with ID starting with 47d9cb75ff89b477399eff78287d48e4c07caabc94d8896aeebad4f8ee7e1b9e not found: ID does not exist" Oct 30 01:03:44.755657 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.755602 5300 scope.go:122] "RemoveContainer" containerID="1e113a42d2dfcf986fd0729b1fbcd06df362bb879d1b833c9737495086637a6a" Oct 30 01:03:44.765209 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.765189 5300 scope.go:122] "RemoveContainer" containerID="1e113a42d2dfcf986fd0729b1fbcd06df362bb879d1b833c9737495086637a6a" Oct 30 01:03:44.765496 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:03:44.765466 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e113a42d2dfcf986fd0729b1fbcd06df362bb879d1b833c9737495086637a6a\": container with ID starting with 1e113a42d2dfcf986fd0729b1fbcd06df362bb879d1b833c9737495086637a6a not found: ID does not exist" containerID="1e113a42d2dfcf986fd0729b1fbcd06df362bb879d1b833c9737495086637a6a" Oct 30 01:03:44.765589 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.765506 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e113a42d2dfcf986fd0729b1fbcd06df362bb879d1b833c9737495086637a6a"} err="failed to get container status \"1e113a42d2dfcf986fd0729b1fbcd06df362bb879d1b833c9737495086637a6a\": rpc error: code = NotFound desc = could not find container \"1e113a42d2dfcf986fd0729b1fbcd06df362bb879d1b833c9737495086637a6a\": container with ID starting with 1e113a42d2dfcf986fd0729b1fbcd06df362bb879d1b833c9737495086637a6a not found: ID does not exist" Oct 30 01:03:44.800976 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.800945 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t"] Oct 30 01:03:44.806568 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.806505 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-837d7038-6c886b9b8c-sql7t"] Oct 30 01:03:44.830650 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.830621 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snnm7\" (UniqueName: \"kubernetes.io/projected/e774204f-594a-4ec4-8232-f318fcca84e8-kube-api-access-snnm7\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:03:44.830650 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.830647 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e774204f-594a-4ec4-8232-f318fcca84e8-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:03:44.845913 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.845874 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm"] Oct 30 01:03:44.848669 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:44.848646 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-837d7038-569cf58996-hz8jm"] Oct 30 01:03:45.821077 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:45.821045 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f84b76-4029-4f4d-aebf-34103ec9ca17" path="/var/lib/kubelet/pods/a5f84b76-4029-4f4d-aebf-34103ec9ca17/volumes" Oct 30 01:03:45.821602 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:45.821584 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e774204f-594a-4ec4-8232-f318fcca84e8" path="/var/lib/kubelet/pods/e774204f-594a-4ec4-8232-f318fcca84e8/volumes" Oct 30 01:03:46.140160 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:46.140076 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:47.788845 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:47.788811 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:49.576920 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:49.576891 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:50.103423 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:50.103390 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:50.571036 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:50.570975 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:50.659673 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:50.659637 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:51.242648 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:51.242605 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:51.592742 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:03:51.592651 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:03:52.703065 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:52.703034 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:53.318105 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:53.318066 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:53.455209 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:53.455173 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:53.538020 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:53.537991 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:53.676151 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:53.676051 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:54.712360 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:54.712323 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:54.813187 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:54.813151 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:54.915228 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:54.915198 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:55.232840 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:55.232807 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:55.557476 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:55.557385 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:56.348544 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:56.348512 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:56.787584 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:56.787551 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:56.884022 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:56.883517 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:56.991367 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:56.991331 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:56.993993 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:56.993969 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk"] Oct 30 01:03:56.994526 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:56.994514 5300 cpu_manager.go:414] "containerMap: removing container" podUID="a5f84b76-4029-4f4d-aebf-34103ec9ca17" containerName="payload-processing" Oct 30 01:03:56.994569 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:56.994528 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f84b76-4029-4f4d-aebf-34103ec9ca17" containerName="payload-processing" Oct 30 01:03:56.994569 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:56.994541 5300 cpu_manager.go:414] "containerMap: removing container" podUID="e81c86f2-0133-4497-b427-798427782cb6" containerName="payload-pre-processing" Oct 30 01:03:56.994569 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:56.994546 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e81c86f2-0133-4497-b427-798427782cb6" containerName="payload-pre-processing" Oct 30 01:03:56.994666 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:56.994584 5300 cpu_manager.go:414] "containerMap: removing container" podUID="e774204f-594a-4ec4-8232-f318fcca84e8" containerName="payload-pre-processing" Oct 30 01:03:56.994666 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:56.994590 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e774204f-594a-4ec4-8232-f318fcca84e8" containerName="payload-pre-processing" Oct 30 01:03:56.994726 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:56.994671 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="e774204f-594a-4ec4-8232-f318fcca84e8" containerName="payload-pre-processing" Oct 30 01:03:56.994726 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:56.994682 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f84b76-4029-4f4d-aebf-34103ec9ca17" containerName="payload-processing" Oct 30 01:03:56.994726 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:56.994688 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="e81c86f2-0133-4497-b427-798427782cb6" containerName="payload-pre-processing" Oct 30 01:03:56.999545 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:56.999521 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk" Oct 30 01:03:57.017220 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:57.017194 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk"] Oct 30 01:03:57.027953 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:57.026930 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-18a189\"" Oct 30 01:03:57.027953 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:57.027052 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-18a189-dockercfg-xbmxx\"" Oct 30 01:03:57.116734 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:57.116636 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgrv5\" (UniqueName: \"kubernetes.io/projected/6b19732a-da65-4b29-a2e7-a2017ce9b815-kube-api-access-lgrv5\") pod \"payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk\" (UID: \"6b19732a-da65-4b29-a2e7-a2017ce9b815\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk" Oct 30 01:03:57.116956 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:57.116862 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b19732a-da65-4b29-a2e7-a2017ce9b815-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk\" (UID: \"6b19732a-da65-4b29-a2e7-a2017ce9b815\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk" Oct 30 01:03:57.218099 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:57.218062 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b19732a-da65-4b29-a2e7-a2017ce9b815-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk\" (UID: \"6b19732a-da65-4b29-a2e7-a2017ce9b815\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk" Oct 30 01:03:57.218270 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:57.218185 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgrv5\" (UniqueName: \"kubernetes.io/projected/6b19732a-da65-4b29-a2e7-a2017ce9b815-kube-api-access-lgrv5\") pod \"payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk\" (UID: \"6b19732a-da65-4b29-a2e7-a2017ce9b815\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk" Oct 30 01:03:57.218864 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:57.218840 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b19732a-da65-4b29-a2e7-a2017ce9b815-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk\" (UID: \"6b19732a-da65-4b29-a2e7-a2017ce9b815\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk" Oct 30 01:03:57.239458 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:57.239427 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgrv5\" (UniqueName: \"kubernetes.io/projected/6b19732a-da65-4b29-a2e7-a2017ce9b815-kube-api-access-lgrv5\") pod \"payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk\" (UID: \"6b19732a-da65-4b29-a2e7-a2017ce9b815\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk" Oct 30 01:03:57.313478 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:57.313438 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk" Oct 30 01:03:57.456429 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:57.456385 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:57.621170 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:57.621140 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk"] Oct 30 01:03:57.621985 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:03:57.621923 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b19732a_da65_4b29_a2e7_a2017ce9b815.slice/crio-716c65e99388430997cb21491a77918f2499a42734985c9a352d76b3442018cc WatchSource:0}: Error finding container 716c65e99388430997cb21491a77918f2499a42734985c9a352d76b3442018cc: Status 404 returned error can't find the container with id 716c65e99388430997cb21491a77918f2499a42734985c9a352d76b3442018cc Oct 30 01:03:57.672137 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:57.672101 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:57.892044 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:57.891947 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk" event={"ID":"6b19732a-da65-4b29-a2e7-a2017ce9b815","Type":"ContainerStarted","Data":"5c00e6157e657fe49107d18a97654b150248268ce3896e00e5d8b3ea63cde9cf"} Oct 30 01:03:57.892044 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:57.892023 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk" event={"ID":"6b19732a-da65-4b29-a2e7-a2017ce9b815","Type":"ContainerStarted","Data":"716c65e99388430997cb21491a77918f2499a42734985c9a352d76b3442018cc"} Oct 30 01:03:57.892232 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:57.892056 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk" Oct 30 01:03:57.934436 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:57.934384 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk" podStartSLOduration=1.9343700419999998 podStartE2EDuration="1.934370042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:57.933937537 +0000 UTC m=+1238.249805845" watchObservedRunningTime="2026-10-30 01:03:57.934370042 +0000 UTC m=+1238.250238366" Oct 30 01:03:58.130059 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:58.130025 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:58.284658 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:58.284625 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:58.912014 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:58.911979 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6"] Oct 30 01:03:58.917221 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:58.917192 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:58.940915 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:58.927709 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6"] Oct 30 01:03:58.940915 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:58.938626 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-fb3370cb-openshift-default-dockercfg-pnjnh\"" Oct 30 01:03:58.983800 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:58.973685 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:59.053268 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.053235 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-podinfo\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.053465 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.053291 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07ae9aeb-afa1-4307-a062-6e2ddb055395-istiod-ca-cert\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.053530 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.053472 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2vwn\" (UniqueName: \"kubernetes.io/projected/07ae9aeb-afa1-4307-a062-6e2ddb055395-kube-api-access-x2vwn\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.053530 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.053501 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-envoy\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.053629 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.053561 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-token\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.053686 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.053654 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-workload-certs\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.053804 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.053753 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-data\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.053899 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.053877 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-workload-socket\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.053965 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.053935 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-credential-socket\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.155242 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.155196 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-podinfo\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.155455 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.155270 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07ae9aeb-afa1-4307-a062-6e2ddb055395-istiod-ca-cert\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.155455 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.155386 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vwn\" (UniqueName: \"kubernetes.io/projected/07ae9aeb-afa1-4307-a062-6e2ddb055395-kube-api-access-x2vwn\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.155455 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.155415 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-envoy\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.155629 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.155461 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-token\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.155629 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.155564 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-workload-certs\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.155629 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.155596 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-data\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.155826 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.155677 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-workload-socket\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.155826 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.155725 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-credential-socket\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.155976 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.155937 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-workload-certs\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.156039 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.155974 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-data\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.156102 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.156075 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07ae9aeb-afa1-4307-a062-6e2ddb055395-istiod-ca-cert\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.156161 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.156078 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-workload-socket\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.156215 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.156179 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-credential-socket\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.157815 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.157793 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-envoy\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.158046 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.158030 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-podinfo\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.178091 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.178018 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2vwn\" (UniqueName: \"kubernetes.io/projected/07ae9aeb-afa1-4307-a062-6e2ddb055395-kube-api-access-x2vwn\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.185017 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.184988 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-token\") pod \"e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.257279 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.257246 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:03:59.257467 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.257376 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:59.380819 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.380743 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:03:59.594067 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.594030 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6"] Oct 30 01:03:59.594824 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:03:59.594789 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ae9aeb_afa1_4307_a062_6e2ddb055395.slice/crio-daafc4b11973d686946fd920c72fb407f62ea1ad08bc2cbc935da70e405a96ac WatchSource:0}: Error finding container daafc4b11973d686946fd920c72fb407f62ea1ad08bc2cbc935da70e405a96ac: Status 404 returned error can't find the container with id daafc4b11973d686946fd920c72fb407f62ea1ad08bc2cbc935da70e405a96ac Oct 30 01:03:59.598032 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.597992 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:03:59.598156 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.598109 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:03:59.598225 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.598154 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:03:59.940585 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.940490 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" event={"ID":"07ae9aeb-afa1-4307-a062-6e2ddb055395","Type":"ContainerStarted","Data":"e428eed35e85bdb6820df1eb011e78408604865ea73b8515d5fa83ebe5ee6ec2"} Oct 30 01:03:59.940585 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.940553 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" event={"ID":"07ae9aeb-afa1-4307-a062-6e2ddb055395","Type":"ContainerStarted","Data":"daafc4b11973d686946fd920c72fb407f62ea1ad08bc2cbc935da70e405a96ac"} Oct 30 01:03:59.988921 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:03:59.986816 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" podStartSLOduration=1.986508802 podStartE2EDuration="1.986508802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:59.984833492 +0000 UTC m=+1240.300701844" watchObservedRunningTime="2026-10-30 01:03:59.986508802 +0000 UTC m=+1240.302377118" Oct 30 01:04:00.257510 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:00.257472 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:04:00.264097 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:00.264067 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:04:00.633794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:00.633579 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:00.739274 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:00.739230 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:00.973196 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:00.973045 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:04:00.974520 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:00.974500 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:04:01.285138 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:01.285050 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:02.404455 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:02.404419 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:02.501391 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:02.501364 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:02.607083 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:02.607049 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:02.727393 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:02.727353 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:03.241059 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:03.241030 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:03.499946 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:03.499856 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:03.598837 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:03.598793 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:04.492926 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:04.492892 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:04.521307 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:04.521271 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7"] Oct 30 01:04:04.525828 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:04.525805 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7" Oct 30 01:04:04.556478 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:04.556444 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fb3370cb\"" Oct 30 01:04:04.561542 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:04.561512 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7"] Oct 30 01:04:04.578200 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:04.578165 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb97f33-12e0-4e9c-8e7c-842363ff915e-plugins-config-volume\") pod \"payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7\" (UID: \"abb97f33-12e0-4e9c-8e7c-842363ff915e\") " pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7" Oct 30 01:04:04.578387 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:04.578220 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf5gg\" (UniqueName: \"kubernetes.io/projected/abb97f33-12e0-4e9c-8e7c-842363ff915e-kube-api-access-rf5gg\") pod \"payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7\" (UID: \"abb97f33-12e0-4e9c-8e7c-842363ff915e\") " pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7" Oct 30 01:04:04.597784 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:04.597731 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:04.689153 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:04.689102 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb97f33-12e0-4e9c-8e7c-842363ff915e-plugins-config-volume\") pod \"payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7\" (UID: \"abb97f33-12e0-4e9c-8e7c-842363ff915e\") " pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7" Oct 30 01:04:04.690662 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:04.690635 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf5gg\" (UniqueName: \"kubernetes.io/projected/abb97f33-12e0-4e9c-8e7c-842363ff915e-kube-api-access-rf5gg\") pod \"payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7\" (UID: \"abb97f33-12e0-4e9c-8e7c-842363ff915e\") " pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7" Oct 30 01:04:04.691444 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:04.690560 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb97f33-12e0-4e9c-8e7c-842363ff915e-plugins-config-volume\") pod \"payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7\" (UID: \"abb97f33-12e0-4e9c-8e7c-842363ff915e\") " pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7" Oct 30 01:04:04.739849 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:04.738113 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf5gg\" (UniqueName: \"kubernetes.io/projected/abb97f33-12e0-4e9c-8e7c-842363ff915e-kube-api-access-rf5gg\") pod \"payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7\" (UID: \"abb97f33-12e0-4e9c-8e7c-842363ff915e\") " pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7" Oct 30 01:04:04.841347 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:04.841243 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7" Oct 30 01:04:05.026081 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:05.026041 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:05.149951 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:05.149912 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7"] Oct 30 01:04:05.150172 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:04:05.150142 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb97f33_12e0_4e9c_8e7c_842363ff915e.slice/crio-b13e3957ef9f4b8fdf2c9e6dd716f593ef42c8b695ec7e830b098fc01e6a415f WatchSource:0}: Error finding container b13e3957ef9f4b8fdf2c9e6dd716f593ef42c8b695ec7e830b098fc01e6a415f: Status 404 returned error can't find the container with id b13e3957ef9f4b8fdf2c9e6dd716f593ef42c8b695ec7e830b098fc01e6a415f Oct 30 01:04:05.292135 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:05.292102 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:05.490260 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:05.490218 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:05.593662 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:05.593621 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:05.705641 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:05.705603 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l"] Oct 30 01:04:05.709387 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:05.709368 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l" Oct 30 01:04:05.727843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:05.727815 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l"] Oct 30 01:04:05.738454 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:05.738424 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-eacea3\"" Oct 30 01:04:05.738609 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:05.738455 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-eacea3-dockercfg-9qpqp\"" Oct 30 01:04:05.785969 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:05.785870 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq"] Oct 30 01:04:05.789935 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:05.789903 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq" Oct 30 01:04:05.817786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:05.816306 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d688cb-3c2f-4d2d-84d6-1dbadcdb2985-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq\" (UID: \"43d688cb-3c2f-4d2d-84d6-1dbadcdb2985\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq" Oct 30 01:04:05.817786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:05.816383 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fj95\" (UniqueName: \"kubernetes.io/projected/43d688cb-3c2f-4d2d-84d6-1dbadcdb2985-kube-api-access-7fj95\") pod \"payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq\" (UID: \"43d688cb-3c2f-4d2d-84d6-1dbadcdb2985\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq" Oct 30 01:04:05.817786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:05.816418 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f667d6d6-df98-49e9-9d07-c9ae6e692863-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l\" (UID: \"f667d6d6-df98-49e9-9d07-c9ae6e692863\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l" Oct 30 01:04:05.817786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:05.816613 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftgjs\" (UniqueName: \"kubernetes.io/projected/f667d6d6-df98-49e9-9d07-c9ae6e692863-kube-api-access-ftgjs\") pod \"payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l\" (UID: \"f667d6d6-df98-49e9-9d07-c9ae6e692863\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l" Oct 30 01:04:05.839808 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:05.826808 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq"] Oct 30 01:04:05.917601 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:05.917560 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftgjs\" (UniqueName: \"kubernetes.io/projected/f667d6d6-df98-49e9-9d07-c9ae6e692863-kube-api-access-ftgjs\") pod \"payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l\" (UID: \"f667d6d6-df98-49e9-9d07-c9ae6e692863\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l" Oct 30 01:04:05.918050 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:05.917740 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d688cb-3c2f-4d2d-84d6-1dbadcdb2985-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq\" (UID: \"43d688cb-3c2f-4d2d-84d6-1dbadcdb2985\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq" Oct 30 01:04:05.918050 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:05.917831 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fj95\" (UniqueName: \"kubernetes.io/projected/43d688cb-3c2f-4d2d-84d6-1dbadcdb2985-kube-api-access-7fj95\") pod \"payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq\" (UID: \"43d688cb-3c2f-4d2d-84d6-1dbadcdb2985\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq" Oct 30 01:04:05.918050 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:05.917864 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f667d6d6-df98-49e9-9d07-c9ae6e692863-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l\" (UID: \"f667d6d6-df98-49e9-9d07-c9ae6e692863\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l" Oct 30 01:04:05.919250 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:05.919215 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f667d6d6-df98-49e9-9d07-c9ae6e692863-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l\" (UID: \"f667d6d6-df98-49e9-9d07-c9ae6e692863\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l" Oct 30 01:04:05.922429 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:05.921602 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d688cb-3c2f-4d2d-84d6-1dbadcdb2985-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq\" (UID: \"43d688cb-3c2f-4d2d-84d6-1dbadcdb2985\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq" Oct 30 01:04:05.940030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:05.939999 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fj95\" (UniqueName: \"kubernetes.io/projected/43d688cb-3c2f-4d2d-84d6-1dbadcdb2985-kube-api-access-7fj95\") pod \"payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq\" (UID: \"43d688cb-3c2f-4d2d-84d6-1dbadcdb2985\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq" Oct 30 01:04:05.940164 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:05.940090 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftgjs\" (UniqueName: \"kubernetes.io/projected/f667d6d6-df98-49e9-9d07-c9ae6e692863-kube-api-access-ftgjs\") pod \"payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l\" (UID: \"f667d6d6-df98-49e9-9d07-c9ae6e692863\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l" Oct 30 01:04:06.022804 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:06.017070 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7" event={"ID":"abb97f33-12e0-4e9c-8e7c-842363ff915e","Type":"ContainerStarted","Data":"8c77b00e86748a3c81eac2c5c7548effc58cdf5f4872a515b16e7b60a3ae0229"} Oct 30 01:04:06.022804 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:06.017142 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7" event={"ID":"abb97f33-12e0-4e9c-8e7c-842363ff915e","Type":"ContainerStarted","Data":"b13e3957ef9f4b8fdf2c9e6dd716f593ef42c8b695ec7e830b098fc01e6a415f"} Oct 30 01:04:06.022804 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:06.017283 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7" Oct 30 01:04:06.023463 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:06.023426 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l" Oct 30 01:04:06.064317 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:06.064255 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7" podStartSLOduration=2.064234725 podStartE2EDuration="2.064234725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:06.062046416 +0000 UTC m=+1246.377914753" watchObservedRunningTime="2026-10-30 01:04:06.064234725 +0000 UTC m=+1246.380103053" Oct 30 01:04:06.115144 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:06.115121 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq" Oct 30 01:04:06.191952 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:06.191849 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:06.349158 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:06.349129 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l"] Oct 30 01:04:06.350895 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:04:06.350868 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf667d6d6_df98_49e9_9d07_c9ae6e692863.slice/crio-25979929eba14421c6d76f822a5f5460a0d409606307cfcc9a03c5473307bc1d WatchSource:0}: Error finding container 25979929eba14421c6d76f822a5f5460a0d409606307cfcc9a03c5473307bc1d: Status 404 returned error can't find the container with id 25979929eba14421c6d76f822a5f5460a0d409606307cfcc9a03c5473307bc1d Oct 30 01:04:06.462988 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:06.462957 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq"] Oct 30 01:04:06.463111 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:04:06.463079 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d688cb_3c2f_4d2d_84d6_1dbadcdb2985.slice/crio-824a7a2049a63de634ca537984284b8dac62f689c63aee462e38f2f9772cabf2 WatchSource:0}: Error finding container 824a7a2049a63de634ca537984284b8dac62f689c63aee462e38f2f9772cabf2: Status 404 returned error can't find the container with id 824a7a2049a63de634ca537984284b8dac62f689c63aee462e38f2f9772cabf2 Oct 30 01:04:07.025098 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:07.025065 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq" event={"ID":"43d688cb-3c2f-4d2d-84d6-1dbadcdb2985","Type":"ContainerStarted","Data":"024b06f5187ad8ee39f85879f1cb251b25cebd771f9cb7648dcaa366283cd0e1"} Oct 30 01:04:07.025571 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:07.025130 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq" Oct 30 01:04:07.025571 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:07.025157 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq" event={"ID":"43d688cb-3c2f-4d2d-84d6-1dbadcdb2985","Type":"ContainerStarted","Data":"824a7a2049a63de634ca537984284b8dac62f689c63aee462e38f2f9772cabf2"} Oct 30 01:04:07.026688 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:07.026661 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l" event={"ID":"f667d6d6-df98-49e9-9d07-c9ae6e692863","Type":"ContainerStarted","Data":"14e04700662f3a2cf006ee4e72e48182b4f1e30f2397ea3944707611de7ba3bd"} Oct 30 01:04:07.026822 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:07.026700 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l" event={"ID":"f667d6d6-df98-49e9-9d07-c9ae6e692863","Type":"ContainerStarted","Data":"25979929eba14421c6d76f822a5f5460a0d409606307cfcc9a03c5473307bc1d"} Oct 30 01:04:07.026822 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:07.026737 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l" Oct 30 01:04:07.033822 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:07.033798 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:07.075077 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:07.075023 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq" podStartSLOduration=2.075009349 podStartE2EDuration="2.075009349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:07.073572462 +0000 UTC m=+1247.389440787" watchObservedRunningTime="2026-10-30 01:04:07.075009349 +0000 UTC m=+1247.390877675" Oct 30 01:04:07.142159 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:07.142076 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l" podStartSLOduration=2.142057352 podStartE2EDuration="2.142057352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:07.135624153 +0000 UTC m=+1247.451492480" watchObservedRunningTime="2026-10-30 01:04:07.142057352 +0000 UTC m=+1247.457925678" Oct 30 01:04:08.746294 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:08.746244 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:08.819931 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:08.819882 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:08.935825 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:08.935798 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk" Oct 30 01:04:09.038727 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:09.032724 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:10.526521 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:10.526469 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:12.189718 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:12.189674 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:13.618943 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:13.618902 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:13.725703 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:13.725671 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:13.821220 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:13.821187 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:14.047865 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:14.047812 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:14.779945 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:14.779912 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:14.878120 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:14.878090 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:14.978181 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:14.978138 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:15.921397 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:15.921347 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:17.029841 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:17.029813 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7" Oct 30 01:04:17.520513 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:17.520466 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm"] Oct 30 01:04:17.525022 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:17.524990 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm" Oct 30 01:04:17.559968 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:17.559937 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-08065c8e-serving-cert\"" Oct 30 01:04:17.562045 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:17.562016 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm"] Oct 30 01:04:17.589361 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:17.589330 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:17.666995 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:17.666966 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:17.714789 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:17.714740 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-openshift-service-ca\") pod \"maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm\" (UID: \"c817200c-b0bc-4fb9-a685-d2c7c08da0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm" Oct 30 01:04:17.715101 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:17.715072 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-maas-api-tls\") pod \"maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm\" (UID: \"c817200c-b0bc-4fb9-a685-d2c7c08da0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm" Oct 30 01:04:17.715101 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:17.715106 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-metrics-certs\") pod \"maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm\" (UID: \"c817200c-b0bc-4fb9-a685-d2c7c08da0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm" Oct 30 01:04:17.715297 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:17.715230 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6km7n\" (UniqueName: \"kubernetes.io/projected/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-kube-api-access-6km7n\") pod \"maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm\" (UID: \"c817200c-b0bc-4fb9-a685-d2c7c08da0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm" Oct 30 01:04:17.767891 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:17.767853 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:17.816909 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:17.816807 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-openshift-service-ca\") pod \"maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm\" (UID: \"c817200c-b0bc-4fb9-a685-d2c7c08da0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm" Oct 30 01:04:17.816909 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:17.816897 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-maas-api-tls\") pod \"maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm\" (UID: \"c817200c-b0bc-4fb9-a685-d2c7c08da0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm" Oct 30 01:04:17.817144 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:17.816929 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-metrics-certs\") pod \"maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm\" (UID: \"c817200c-b0bc-4fb9-a685-d2c7c08da0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm" Oct 30 01:04:17.817144 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:17.817032 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6km7n\" (UniqueName: \"kubernetes.io/projected/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-kube-api-access-6km7n\") pod \"maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm\" (UID: \"c817200c-b0bc-4fb9-a685-d2c7c08da0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm" Oct 30 01:04:17.818208 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:17.818179 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-openshift-service-ca\") pod \"maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm\" (UID: \"c817200c-b0bc-4fb9-a685-d2c7c08da0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm" Oct 30 01:04:17.825314 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:17.825131 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-metrics-certs\") pod \"maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm\" (UID: \"c817200c-b0bc-4fb9-a685-d2c7c08da0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm" Oct 30 01:04:17.825504 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:17.825479 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-maas-api-tls\") pod \"maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm\" (UID: \"c817200c-b0bc-4fb9-a685-d2c7c08da0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm" Oct 30 01:04:17.838816 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:17.838789 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6km7n\" (UniqueName: \"kubernetes.io/projected/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-kube-api-access-6km7n\") pod \"maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm\" (UID: \"c817200c-b0bc-4fb9-a685-d2c7c08da0a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm" Oct 30 01:04:17.840634 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:17.840609 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm" Oct 30 01:04:18.036900 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:18.036871 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq" Oct 30 01:04:18.037367 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:18.036944 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l" Oct 30 01:04:18.171412 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:18.171377 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm"] Oct 30 01:04:18.173012 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:04:18.172975 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc817200c_b0bc_4fb9_a685_d2c7c08da0a5.slice/crio-0ef0955851923f4b6215172677611b7f01e7a257c035b2c36350d43786df9194 WatchSource:0}: Error finding container 0ef0955851923f4b6215172677611b7f01e7a257c035b2c36350d43786df9194: Status 404 returned error can't find the container with id 0ef0955851923f4b6215172677611b7f01e7a257c035b2c36350d43786df9194 Oct 30 01:04:19.126519 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:19.126483 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm" event={"ID":"c817200c-b0bc-4fb9-a685-d2c7c08da0a5","Type":"ContainerStarted","Data":"84bd857d54ab5021f373d572b1e935c87934a5435e30accfb9cbb89477895b73"} Oct 30 01:04:19.127023 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:19.126540 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm" event={"ID":"c817200c-b0bc-4fb9-a685-d2c7c08da0a5","Type":"ContainerStarted","Data":"0ef0955851923f4b6215172677611b7f01e7a257c035b2c36350d43786df9194"} Oct 30 01:04:19.127023 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:19.126563 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm" Oct 30 01:04:19.169854 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:19.169798 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm" podStartSLOduration=1.853488684 podStartE2EDuration="2.169782989s" totalImagesPullingTime="316.294305ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:19.168311402 +0000 UTC m=+1259.484179729" watchObservedRunningTime="2026-10-30 01:04:19.169782989 +0000 UTC m=+1259.485651313" Oct 30 01:04:19.279286 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:19.279252 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:21.022106 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.022068 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx"] Oct 30 01:04:21.028072 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.028052 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx" Oct 30 01:04:21.044044 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.044013 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx"] Oct 30 01:04:21.054095 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.054067 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fb3370cb-serving-cert\"" Oct 30 01:04:21.075456 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.075424 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp74t\" (UniqueName: \"kubernetes.io/projected/f34da3d1-b871-4daa-b86a-5376585b1c1e-kube-api-access-rp74t\") pod \"maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx\" (UID: \"f34da3d1-b871-4daa-b86a-5376585b1c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx" Oct 30 01:04:21.075642 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.075473 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f34da3d1-b871-4daa-b86a-5376585b1c1e-openshift-service-ca\") pod \"maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx\" (UID: \"f34da3d1-b871-4daa-b86a-5376585b1c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx" Oct 30 01:04:21.075642 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.075588 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f34da3d1-b871-4daa-b86a-5376585b1c1e-maas-api-tls\") pod \"maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx\" (UID: \"f34da3d1-b871-4daa-b86a-5376585b1c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx" Oct 30 01:04:21.075814 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.075795 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f34da3d1-b871-4daa-b86a-5376585b1c1e-metrics-certs\") pod \"maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx\" (UID: \"f34da3d1-b871-4daa-b86a-5376585b1c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx" Oct 30 01:04:21.093184 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.093149 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:21.171609 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.171576 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:21.177269 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.177229 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f34da3d1-b871-4daa-b86a-5376585b1c1e-metrics-certs\") pod \"maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx\" (UID: \"f34da3d1-b871-4daa-b86a-5376585b1c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx" Oct 30 01:04:21.177398 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.177374 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp74t\" (UniqueName: \"kubernetes.io/projected/f34da3d1-b871-4daa-b86a-5376585b1c1e-kube-api-access-rp74t\") pod \"maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx\" (UID: \"f34da3d1-b871-4daa-b86a-5376585b1c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx" Oct 30 01:04:21.177485 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.177410 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f34da3d1-b871-4daa-b86a-5376585b1c1e-openshift-service-ca\") pod \"maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx\" (UID: \"f34da3d1-b871-4daa-b86a-5376585b1c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx" Oct 30 01:04:21.177485 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.177478 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f34da3d1-b871-4daa-b86a-5376585b1c1e-maas-api-tls\") pod \"maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx\" (UID: \"f34da3d1-b871-4daa-b86a-5376585b1c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx" Oct 30 01:04:21.177638 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:04:21.177622 5300 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-serving-cert: secret "maas-api-e2e-mt-fb3370cb-serving-cert" not found Oct 30 01:04:21.177703 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:04:21.177693 5300 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f34da3d1-b871-4daa-b86a-5376585b1c1e-maas-api-tls podName:f34da3d1-b871-4daa-b86a-5376585b1c1e nodeName:}" failed. No retries permitted until 2026-10-30 01:04:21.677671864 +0000 UTC m=+1261.993540182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f34da3d1-b871-4daa-b86a-5376585b1c1e-maas-api-tls") pod "maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx" (UID: "f34da3d1-b871-4daa-b86a-5376585b1c1e") : secret "maas-api-e2e-mt-fb3370cb-serving-cert" not found Oct 30 01:04:21.178157 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.178136 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f34da3d1-b871-4daa-b86a-5376585b1c1e-openshift-service-ca\") pod \"maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx\" (UID: \"f34da3d1-b871-4daa-b86a-5376585b1c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx" Oct 30 01:04:21.179867 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.179843 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f34da3d1-b871-4daa-b86a-5376585b1c1e-metrics-certs\") pod \"maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx\" (UID: \"f34da3d1-b871-4daa-b86a-5376585b1c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx" Oct 30 01:04:21.198912 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.198884 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp74t\" (UniqueName: \"kubernetes.io/projected/f34da3d1-b871-4daa-b86a-5376585b1c1e-kube-api-access-rp74t\") pod \"maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx\" (UID: \"f34da3d1-b871-4daa-b86a-5376585b1c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx" Oct 30 01:04:21.481149 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.481111 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm"] Oct 30 01:04:21.489455 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.489418 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.510228 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.510156 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm"] Oct 30 01:04:21.514592 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.514562 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft"] Oct 30 01:04:21.525172 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.524138 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.544873 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.544831 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft"] Oct 30 01:04:21.549508 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.549464 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:21.583850 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.583804 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.584043 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.583881 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd004c0-3447-4382-82f5-e797f86b850e-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.584043 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.583974 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.584043 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.584027 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.584230 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.584059 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd004c0-3447-4382-82f5-e797f86b850e-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.584230 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.584115 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded1d999-96c4-4114-a626-615fa35853f2-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.584230 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.584148 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khkkg\" (UniqueName: \"kubernetes.io/projected/ded1d999-96c4-4114-a626-615fa35853f2-kube-api-access-khkkg\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.584380 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.584269 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.584380 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.584346 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9tnd\" (UniqueName: \"kubernetes.io/projected/7bd004c0-3447-4382-82f5-e797f86b850e-kube-api-access-p9tnd\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.584481 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.584425 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.584481 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.584465 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.584585 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.584492 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.584585 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.584517 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.584585 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.584553 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ded1d999-96c4-4114-a626-615fa35853f2-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.584740 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.584591 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd004c0-3447-4382-82f5-e797f86b850e-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.584740 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.584730 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.584870 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.584809 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded1d999-96c4-4114-a626-615fa35853f2-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.584870 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.584847 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.596379 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:04:21.596338 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:04:21.686322 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.686276 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.686535 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.686338 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd004c0-3447-4382-82f5-e797f86b850e-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.686535 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.686368 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.686535 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.686400 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.686535 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.686422 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd004c0-3447-4382-82f5-e797f86b850e-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.686535 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.686460 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f34da3d1-b871-4daa-b86a-5376585b1c1e-maas-api-tls\") pod \"maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx\" (UID: \"f34da3d1-b871-4daa-b86a-5376585b1c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx" Oct 30 01:04:21.686535 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.686486 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded1d999-96c4-4114-a626-615fa35853f2-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.686535 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.686512 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khkkg\" (UniqueName: \"kubernetes.io/projected/ded1d999-96c4-4114-a626-615fa35853f2-kube-api-access-khkkg\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.686954 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.686538 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.686954 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.686599 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9tnd\" (UniqueName: \"kubernetes.io/projected/7bd004c0-3447-4382-82f5-e797f86b850e-kube-api-access-p9tnd\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.686954 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.686641 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.686954 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.686675 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.686954 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.686702 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.686954 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.686726 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.686954 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.686783 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ded1d999-96c4-4114-a626-615fa35853f2-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.686954 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.686826 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd004c0-3447-4382-82f5-e797f86b850e-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.686954 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.686931 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.687422 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.686973 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.687422 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.687042 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded1d999-96c4-4114-a626-615fa35853f2-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.687422 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.687082 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.687422 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.687112 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.687422 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.687298 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.687715 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.687490 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd004c0-3447-4382-82f5-e797f86b850e-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.687715 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.687502 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-workload-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.687715 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.687492 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-credential-socket\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.688034 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.687749 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.688034 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.687957 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-workload-certs\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.688151 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.688133 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded1d999-96c4-4114-a626-615fa35853f2-istiod-ca-cert\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.688238 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.688216 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-istio-data\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.689279 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.689227 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.689684 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.689667 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded1d999-96c4-4114-a626-615fa35853f2-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.691249 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.691225 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-istio-envoy\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.691337 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.691244 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd004c0-3447-4382-82f5-e797f86b850e-istio-podinfo\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.691380 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.691353 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f34da3d1-b871-4daa-b86a-5376585b1c1e-maas-api-tls\") pod \"maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx\" (UID: \"f34da3d1-b871-4daa-b86a-5376585b1c1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx" Oct 30 01:04:21.708664 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.708635 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ded1d999-96c4-4114-a626-615fa35853f2-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.708836 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.708811 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd004c0-3447-4382-82f5-e797f86b850e-istio-token\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.708948 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.708930 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9tnd\" (UniqueName: \"kubernetes.io/projected/7bd004c0-3447-4382-82f5-e797f86b850e-kube-api-access-p9tnd\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.709293 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.709278 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khkkg\" (UniqueName: \"kubernetes.io/projected/ded1d999-96c4-4114-a626-615fa35853f2-kube-api-access-khkkg\") pod \"e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.808583 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.808495 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:21.853986 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.853953 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:21.970673 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:21.970638 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx" Oct 30 01:04:22.138597 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:22.138557 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm"] Oct 30 01:04:22.139892 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:04:22.139863 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd004c0_3447_4382_82f5_e797f86b850e.slice/crio-8a7603eda0b57f931831c65f8db5ad23ea8715afcd55126b5eafc0576a5737f6 WatchSource:0}: Error finding container 8a7603eda0b57f931831c65f8db5ad23ea8715afcd55126b5eafc0576a5737f6: Status 404 returned error can't find the container with id 8a7603eda0b57f931831c65f8db5ad23ea8715afcd55126b5eafc0576a5737f6 Oct 30 01:04:22.142824 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:22.142792 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:04:22.142921 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:22.142863 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:04:22.142921 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:22.142904 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:04:22.148559 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:22.148530 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" event={"ID":"7bd004c0-3447-4382-82f5-e797f86b850e","Type":"ContainerStarted","Data":"8a7603eda0b57f931831c65f8db5ad23ea8715afcd55126b5eafc0576a5737f6"} Oct 30 01:04:22.190231 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:22.190204 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft"] Oct 30 01:04:22.191276 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:04:22.191249 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded1d999_96c4_4114_a626_615fa35853f2.slice/crio-cb5e388c4fadb6e694f6c15d1d948408a8392a1a40a026e1315a406ceaf4274a WatchSource:0}: Error finding container cb5e388c4fadb6e694f6c15d1d948408a8392a1a40a026e1315a406ceaf4274a: Status 404 returned error can't find the container with id cb5e388c4fadb6e694f6c15d1d948408a8392a1a40a026e1315a406ceaf4274a Oct 30 01:04:22.194632 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:22.194599 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:04:22.194685 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:22.194671 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:04:22.194717 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:22.194704 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:04:22.340365 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:22.339947 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx"] Oct 30 01:04:22.915228 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:22.915131 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:23.012801 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:23.012748 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:23.187326 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:23.187240 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" event={"ID":"7bd004c0-3447-4382-82f5-e797f86b850e","Type":"ContainerStarted","Data":"017ed38f55ce4b5a817d8e5863c39dafb727cc66d33f24b33d6bb300359b470b"} Oct 30 01:04:23.198784 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:23.197907 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" event={"ID":"ded1d999-96c4-4114-a626-615fa35853f2","Type":"ContainerStarted","Data":"e1371e9f1aba1eb8aec5963ef94caac1ebcee0093572c812adcb7c5b1300f01e"} Oct 30 01:04:23.198784 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:23.198001 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" event={"ID":"ded1d999-96c4-4114-a626-615fa35853f2","Type":"ContainerStarted","Data":"cb5e388c4fadb6e694f6c15d1d948408a8392a1a40a026e1315a406ceaf4274a"} Oct 30 01:04:23.203920 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:23.200821 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx" event={"ID":"f34da3d1-b871-4daa-b86a-5376585b1c1e","Type":"ContainerStarted","Data":"405ccbc40350d29f35e7911dc452f8a829207cfa7bd4390f128a36145ff706dc"} Oct 30 01:04:23.203920 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:23.200874 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx" event={"ID":"f34da3d1-b871-4daa-b86a-5376585b1c1e","Type":"ContainerStarted","Data":"04f13af169cf355bcbf6c594ab77f98e0aa74ca2221670b152cc3d277aa0ed3f"} Oct 30 01:04:23.203920 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:23.201442 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx" Oct 30 01:04:23.306027 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:23.305997 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:23.342785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:23.340097 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" podStartSLOduration=2.340073122 podStartE2EDuration="2.340073122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:23.277411964 +0000 UTC m=+1263.593280294" watchObservedRunningTime="2026-10-30 01:04:23.340073122 +0000 UTC m=+1263.655941447" Oct 30 01:04:23.370977 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:23.366025 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" podStartSLOduration=2.366004682 podStartE2EDuration="2.366004682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:23.339117411 +0000 UTC m=+1263.654985740" watchObservedRunningTime="2026-10-30 01:04:23.366004682 +0000 UTC m=+1263.681873009" Oct 30 01:04:23.399555 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:23.399491 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx" podStartSLOduration=3.115848333 podStartE2EDuration="3.39947372s" totalImagesPullingTime="283.625387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:23.39654885 +0000 UTC m=+1263.712417176" watchObservedRunningTime="2026-10-30 01:04:23.39947372 +0000 UTC m=+1263.715342049" Oct 30 01:04:23.809835 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:23.809699 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:23.812020 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:23.811997 5300 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:04:23.812099 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:23.812045 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" podUID="7bd004c0-3447-4382-82f5-e797f86b850e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 01:04:23.877816 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:23.876978 5300 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 01:04:23.877816 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:23.877047 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" podUID="ded1d999-96c4-4114-a626-615fa35853f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 01:04:23.891283 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:23.891254 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:24.809664 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:24.809630 5300 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:04:24.810117 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:24.809685 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" podUID="7bd004c0-3447-4382-82f5-e797f86b850e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 01:04:24.861786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:24.855911 5300 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 01:04:24.861786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:24.855971 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" podUID="ded1d999-96c4-4114-a626-615fa35853f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 01:04:25.008717 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:25.008668 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:25.179204 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:25.179110 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm" Oct 30 01:04:25.810192 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:25.810146 5300 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:04:25.810607 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:25.810227 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" podUID="7bd004c0-3447-4382-82f5-e797f86b850e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 01:04:25.868786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:25.867939 5300 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 01:04:25.868786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:25.867997 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" podUID="ded1d999-96c4-4114-a626-615fa35853f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 01:04:26.809003 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:26.808912 5300 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:04:26.809003 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:26.808971 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" podUID="7bd004c0-3447-4382-82f5-e797f86b850e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 01:04:26.869563 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:26.869527 5300 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 01:04:26.870209 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:26.870164 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" podUID="ded1d999-96c4-4114-a626-615fa35853f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 01:04:27.043799 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:27.043703 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:27.278785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:27.277374 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:27.809142 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:27.809067 5300 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:04:27.809142 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:27.809129 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" podUID="7bd004c0-3447-4382-82f5-e797f86b850e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 01:04:27.871788 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:27.866574 5300 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 01:04:27.871788 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:27.866630 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" podUID="ded1d999-96c4-4114-a626-615fa35853f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 01:04:28.808938 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:28.808898 5300 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:04:28.809250 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:28.809211 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" podUID="7bd004c0-3447-4382-82f5-e797f86b850e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 01:04:28.857507 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:28.857468 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:28.857794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:28.857655 5300 patch_prober.go:29] interesting pod/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 01:04:28.857950 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:28.857921 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" podUID="ded1d999-96c4-4114-a626-615fa35853f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 01:04:29.219829 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:29.219706 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx" Oct 30 01:04:29.823524 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:29.823492 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:29.823808 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:29.823787 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:29.824775 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:29.824733 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:04:29.859057 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:29.859022 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:29.859319 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:29.859300 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:29.860079 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:29.860058 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:04:30.528390 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:30.528353 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:31.315975 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:31.315946 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:32.527486 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:32.527448 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:34.075734 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:34.075701 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:34.339659 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:34.339575 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:34.974023 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:34.973989 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:35.413722 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:35.413638 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:35.882145 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:35.882106 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:35.983428 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:35.983396 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:36.084432 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:36.084384 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:36.856064 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:36.856022 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:36.957128 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:36.957091 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:37.175976 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:37.175888 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:37.786345 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:37.786306 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:39.540886 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:39.540857 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:40.033919 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:40.033875 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:40.151784 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:40.142458 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:41.534649 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:41.534602 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:43.340785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:43.305304 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:43.438072 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:43.437939 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:43.773792 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:43.764186 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:43.951785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:43.950945 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:04:43.951785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:43.951346 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 30 01:04:43.952044 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:43.952013 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:04:43.972191 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:43.952338 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" podUID="d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:04:44.112841 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:44.102755 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:44.393406 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:44.393296 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:44.495233 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:44.495200 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:44.696959 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:44.696854 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 01:04:44.697300 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:44.697264 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 01:04:44.854786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:44.849496 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p" podUID="1166fd98-968e-4fda-a0b9-0cfea1ed9e03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:04:44.854786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:44.851326 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:44.856875 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:44.856834 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:04:44.956843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:44.952883 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" podUID="d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:04:44.956843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:44.953785 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 01:04:44.976845 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:44.976666 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:04:44.977058 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:44.976857 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 30 01:04:44.977201 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:44.977177 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 01:04:44.978209 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:44.978116 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 01:04:44.978316 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:44.978271 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 01:04:44.978395 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:44.978325 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 01:04:45.297496 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:45.297460 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 01:04:45.297827 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:45.297810 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 01:04:45.298735 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:45.298533 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 01:04:45.305968 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:45.299350 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 01:04:46.345488 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:46.345445 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:46.954783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:46.954434 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:47.199374 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:47.199342 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:47.305571 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:47.305519 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:47.582979 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:47.577715 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:47.777542 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:47.777511 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:47.870886 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:47.870795 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:47.971170 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:47.971132 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:48.071179 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:48.071145 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:48.986555 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:48.986506 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:50.890346 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:50.890258 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:51.598917 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:04:51.598877 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:04:52.550254 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:52.550221 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:52.721480 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:52.721440 5300 generic.go:356] "Generic (PLEG): container finished" podID="07ae9aeb-afa1-4307-a062-6e2ddb055395" containerID="e428eed35e85bdb6820df1eb011e78408604865ea73b8515d5fa83ebe5ee6ec2" exitCode=0 Oct 30 01:04:52.721480 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:52.721492 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" event={"ID":"07ae9aeb-afa1-4307-a062-6e2ddb055395","Type":"ContainerDied","Data":"e428eed35e85bdb6820df1eb011e78408604865ea73b8515d5fa83ebe5ee6ec2"} Oct 30 01:04:52.722079 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:52.722059 5300 scope.go:122] "RemoveContainer" containerID="e428eed35e85bdb6820df1eb011e78408604865ea73b8515d5fa83ebe5ee6ec2" Oct 30 01:04:52.723948 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:52.723916 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:04:52.724048 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:52.723991 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:04:52.724048 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:52.724032 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:04:53.756747 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:53.756712 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" event={"ID":"07ae9aeb-afa1-4307-a062-6e2ddb055395","Type":"ContainerStarted","Data":"420951f796b37a9cbe00156a839aa7e9a453842c93ca6164d7d731cedcc1bd84"} Oct 30 01:04:53.946967 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:53.946934 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:54.244555 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:54.244076 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:54.257833 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:54.257802 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:04:54.258032 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:54.258006 5300 patch_prober.go:29] interesting pod/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 01:04:54.258112 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:54.258060 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" podUID="07ae9aeb-afa1-4307-a062-6e2ddb055395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 01:04:54.388459 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:54.388412 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:54.488432 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:54.488395 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:55.186607 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:55.186555 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:55.267310 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:55.263586 5300 patch_prober.go:29] interesting pod/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 01:04:55.267310 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:55.263697 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" podUID="07ae9aeb-afa1-4307-a062-6e2ddb055395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 01:04:55.296445 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:55.296412 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:55.404584 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:55.404552 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:55.514456 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:55.514382 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:55.776193 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:55.776107 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:55.891188 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:55.891146 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:56.150032 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:56.149948 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:56.257730 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:56.257700 5300 patch_prober.go:29] interesting pod/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 01:04:56.258166 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:56.257749 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" podUID="07ae9aeb-afa1-4307-a062-6e2ddb055395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 01:04:56.353246 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:56.353216 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:57.242866 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:57.241208 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:57.257680 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:57.257652 5300 patch_prober.go:29] interesting pod/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 01:04:57.257864 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:57.257700 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" podUID="07ae9aeb-afa1-4307-a062-6e2ddb055395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 01:04:57.995397 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:57.995364 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:58.263415 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:58.263306 5300 patch_prober.go:29] interesting pod/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 01:04:58.263952 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:58.263505 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" podUID="07ae9aeb-afa1-4307-a062-6e2ddb055395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 01:04:58.362465 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:58.362432 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:04:59.261987 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:59.261956 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:04:59.262252 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:59.262229 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:04:59.262856 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:59.262836 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:04:59.743732 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:04:59.743697 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:00.716148 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:00.716092 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:00.815788 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:00.815742 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:00.915230 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:00.915193 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:01.015067 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:01.015033 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:01.892057 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:01.892025 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:01.994420 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:01.994388 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:02.508882 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:02.508847 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:02.608115 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:02.608082 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:02.631346 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:02.631314 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f"] Oct 30 01:05:02.637029 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:02.637008 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f" Oct 30 01:05:02.656829 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:02.656803 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f"] Oct 30 01:05:02.665883 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:02.665857 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-17d855\"" Oct 30 01:05:02.691182 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:02.691157 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-17d855-dockercfg-rqxrs\"" Oct 30 01:05:02.701234 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:02.701206 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257"] Oct 30 01:05:02.705568 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:02.705533 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257" Oct 30 01:05:02.725342 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:02.725318 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257"] Oct 30 01:05:02.733525 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:02.733496 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbkjd\" (UniqueName: \"kubernetes.io/projected/a22c9a38-2f90-46f2-b924-0a28e85ac2fa-kube-api-access-tbkjd\") pod \"payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f\" (UID: \"a22c9a38-2f90-46f2-b924-0a28e85ac2fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f" Oct 30 01:05:02.733729 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:02.733708 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a22c9a38-2f90-46f2-b924-0a28e85ac2fa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f\" (UID: \"a22c9a38-2f90-46f2-b924-0a28e85ac2fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f" Oct 30 01:05:02.866545 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:02.866444 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbkjd\" (UniqueName: \"kubernetes.io/projected/a22c9a38-2f90-46f2-b924-0a28e85ac2fa-kube-api-access-tbkjd\") pod \"payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f\" (UID: \"a22c9a38-2f90-46f2-b924-0a28e85ac2fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f" Oct 30 01:05:02.866830 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:02.866808 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a22c9a38-2f90-46f2-b924-0a28e85ac2fa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f\" (UID: \"a22c9a38-2f90-46f2-b924-0a28e85ac2fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f" Oct 30 01:05:02.867021 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:02.866999 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zjbm\" (UniqueName: \"kubernetes.io/projected/fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37-kube-api-access-8zjbm\") pod \"payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257\" (UID: \"fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257" Oct 30 01:05:02.867302 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:02.867286 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257\" (UID: \"fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257" Oct 30 01:05:02.889671 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:02.870200 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a22c9a38-2f90-46f2-b924-0a28e85ac2fa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f\" (UID: \"a22c9a38-2f90-46f2-b924-0a28e85ac2fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f" Oct 30 01:05:02.906536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:02.893235 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbkjd\" (UniqueName: \"kubernetes.io/projected/a22c9a38-2f90-46f2-b924-0a28e85ac2fa-kube-api-access-tbkjd\") pod \"payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f\" (UID: \"a22c9a38-2f90-46f2-b924-0a28e85ac2fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f" Oct 30 01:05:02.951133 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:02.951080 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f" Oct 30 01:05:02.991508 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:02.991470 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257\" (UID: \"fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257" Oct 30 01:05:02.991777 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:02.991745 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zjbm\" (UniqueName: \"kubernetes.io/projected/fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37-kube-api-access-8zjbm\") pod \"payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257\" (UID: \"fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257" Oct 30 01:05:02.993363 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:02.993336 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257\" (UID: \"fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257" Oct 30 01:05:03.016441 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.016379 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zjbm\" (UniqueName: \"kubernetes.io/projected/fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37-kube-api-access-8zjbm\") pod \"payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257\" (UID: \"fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257" Oct 30 01:05:03.022529 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.022041 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257" Oct 30 01:05:03.261298 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:05:03.261263 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22c9a38_2f90_46f2_b924_0a28e85ac2fa.slice/crio-8c3b49e3c97ceb4eee8b123262df0953029daae863b4fb248ab2f96290c0582c WatchSource:0}: Error finding container 8c3b49e3c97ceb4eee8b123262df0953029daae863b4fb248ab2f96290c0582c: Status 404 returned error can't find the container with id 8c3b49e3c97ceb4eee8b123262df0953029daae863b4fb248ab2f96290c0582c Oct 30 01:05:03.261850 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.261827 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f"] Oct 30 01:05:03.358810 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:05:03.358781 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa4a3b1f_1bd3_4a67_b2c3_f8db8f241d37.slice/crio-613103df30b6730c92af27e0a506e3c80f2deb3882147238a7147fbe73f5c806 WatchSource:0}: Error finding container 613103df30b6730c92af27e0a506e3c80f2deb3882147238a7147fbe73f5c806: Status 404 returned error can't find the container with id 613103df30b6730c92af27e0a506e3c80f2deb3882147238a7147fbe73f5c806 Oct 30 01:05:03.360360 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.360333 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257"] Oct 30 01:05:03.689789 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.689690 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8"] Oct 30 01:05:03.694428 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.694410 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8" Oct 30 01:05:03.706439 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.706413 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f37290a-2d88-43bc-8af1-64bff456c157-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8\" (UID: \"5f37290a-2d88-43bc-8af1-64bff456c157\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8" Oct 30 01:05:03.706566 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.706452 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f37290a-2d88-43bc-8af1-64bff456c157-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8\" (UID: \"5f37290a-2d88-43bc-8af1-64bff456c157\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8" Oct 30 01:05:03.706566 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.706518 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f37290a-2d88-43bc-8af1-64bff456c157-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8\" (UID: \"5f37290a-2d88-43bc-8af1-64bff456c157\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8" Oct 30 01:05:03.706654 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.706625 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzkzh\" (UniqueName: \"kubernetes.io/projected/5f37290a-2d88-43bc-8af1-64bff456c157-kube-api-access-dzkzh\") pod \"maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8\" (UID: \"5f37290a-2d88-43bc-8af1-64bff456c157\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8" Oct 30 01:05:03.712002 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.711979 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-2be695-serving-cert\"" Oct 30 01:05:03.717063 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.717039 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8"] Oct 30 01:05:03.756717 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.756662 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:03.807859 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.807817 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f37290a-2d88-43bc-8af1-64bff456c157-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8\" (UID: \"5f37290a-2d88-43bc-8af1-64bff456c157\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8" Oct 30 01:05:03.808060 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.807883 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f37290a-2d88-43bc-8af1-64bff456c157-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8\" (UID: \"5f37290a-2d88-43bc-8af1-64bff456c157\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8" Oct 30 01:05:03.808060 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.807920 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f37290a-2d88-43bc-8af1-64bff456c157-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8\" (UID: \"5f37290a-2d88-43bc-8af1-64bff456c157\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8" Oct 30 01:05:03.808060 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.807945 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzkzh\" (UniqueName: \"kubernetes.io/projected/5f37290a-2d88-43bc-8af1-64bff456c157-kube-api-access-dzkzh\") pod \"maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8\" (UID: \"5f37290a-2d88-43bc-8af1-64bff456c157\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8" Oct 30 01:05:03.808060 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:05:03.808050 5300 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-serving-cert: secret "maas-api-e2e-shared-b-w2-2be695-serving-cert" not found Oct 30 01:05:03.808281 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:05:03.808119 5300 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f37290a-2d88-43bc-8af1-64bff456c157-maas-api-tls podName:5f37290a-2d88-43bc-8af1-64bff456c157 nodeName:}" failed. No retries permitted until 2026-10-30 01:05:04.308099316 +0000 UTC m=+1304.623967623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5f37290a-2d88-43bc-8af1-64bff456c157-maas-api-tls") pod "maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8" (UID: "5f37290a-2d88-43bc-8af1-64bff456c157") : secret "maas-api-e2e-shared-b-w2-2be695-serving-cert" not found Oct 30 01:05:03.808647 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.808618 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f37290a-2d88-43bc-8af1-64bff456c157-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8\" (UID: \"5f37290a-2d88-43bc-8af1-64bff456c157\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8" Oct 30 01:05:03.810509 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.810487 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f37290a-2d88-43bc-8af1-64bff456c157-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8\" (UID: \"5f37290a-2d88-43bc-8af1-64bff456c157\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8" Oct 30 01:05:03.831809 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.831746 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzkzh\" (UniqueName: \"kubernetes.io/projected/5f37290a-2d88-43bc-8af1-64bff456c157-kube-api-access-dzkzh\") pod \"maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8\" (UID: \"5f37290a-2d88-43bc-8af1-64bff456c157\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8" Oct 30 01:05:03.835090 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.835042 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:03.842112 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.842088 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257" event={"ID":"fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37","Type":"ContainerStarted","Data":"3301c565b1809fb85cad92634ee899d886520b2c2b8ceeb3bd2316e7f945111a"} Oct 30 01:05:03.842255 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.842132 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257" event={"ID":"fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37","Type":"ContainerStarted","Data":"613103df30b6730c92af27e0a506e3c80f2deb3882147238a7147fbe73f5c806"} Oct 30 01:05:03.842317 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.842297 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257" Oct 30 01:05:03.843775 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.843741 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f" event={"ID":"a22c9a38-2f90-46f2-b924-0a28e85ac2fa","Type":"ContainerStarted","Data":"951e706344460305c279f5b991194c967e6163fdea439c2c86266afb73496c7c"} Oct 30 01:05:03.843867 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.843791 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f" event={"ID":"a22c9a38-2f90-46f2-b924-0a28e85ac2fa","Type":"ContainerStarted","Data":"8c3b49e3c97ceb4eee8b123262df0953029daae863b4fb248ab2f96290c0582c"} Oct 30 01:05:03.843867 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.843856 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f" Oct 30 01:05:03.937506 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.932260 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257" podStartSLOduration=1.932241357 podStartE2EDuration="1.932241357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:03.887030378 +0000 UTC m=+1304.202898722" watchObservedRunningTime="2026-10-30 01:05:03.932241357 +0000 UTC m=+1304.248109682" Oct 30 01:05:03.942477 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.940272 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f" podStartSLOduration=1.940257323 podStartE2EDuration="1.940257323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:03.928313801 +0000 UTC m=+1304.244182127" watchObservedRunningTime="2026-10-30 01:05:03.940257323 +0000 UTC m=+1304.256125653" Oct 30 01:05:03.999996 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:03.999961 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:04.321429 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:04.321387 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f37290a-2d88-43bc-8af1-64bff456c157-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8\" (UID: \"5f37290a-2d88-43bc-8af1-64bff456c157\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8" Oct 30 01:05:04.323982 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:04.323956 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f37290a-2d88-43bc-8af1-64bff456c157-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8\" (UID: \"5f37290a-2d88-43bc-8af1-64bff456c157\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8" Oct 30 01:05:04.608950 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:04.608864 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8" Oct 30 01:05:04.950091 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:04.949999 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8"] Oct 30 01:05:04.958578 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:05:04.958548 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f37290a_2d88_43bc_8af1_64bff456c157.slice/crio-db53d71be02ab00b4d4b7838e106c6f2794ea03331bffbcf602f11d2bf54b7f4 WatchSource:0}: Error finding container db53d71be02ab00b4d4b7838e106c6f2794ea03331bffbcf602f11d2bf54b7f4: Status 404 returned error can't find the container with id db53d71be02ab00b4d4b7838e106c6f2794ea03331bffbcf602f11d2bf54b7f4 Oct 30 01:05:05.632184 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:05.632101 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:05.730412 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:05.730382 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:05.862259 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:05.862219 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8" event={"ID":"5f37290a-2d88-43bc-8af1-64bff456c157","Type":"ContainerStarted","Data":"d8427b2be59f630dd354ef1d8e83779122890618f547dc1401ea06ffc14c8486"} Oct 30 01:05:05.862456 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:05.862288 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8" event={"ID":"5f37290a-2d88-43bc-8af1-64bff456c157","Type":"ContainerStarted","Data":"db53d71be02ab00b4d4b7838e106c6f2794ea03331bffbcf602f11d2bf54b7f4"} Oct 30 01:05:05.862456 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:05.862335 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8" Oct 30 01:05:05.927148 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:05.925824 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8" podStartSLOduration=2.6622791599999998 podStartE2EDuration="2.925802476s" totalImagesPullingTime="263.523316ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:05.925000876 +0000 UTC m=+1306.240869204" watchObservedRunningTime="2026-10-30 01:05:05.925802476 +0000 UTC m=+1306.241670820" Oct 30 01:05:05.968090 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:05.967635 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db"] Oct 30 01:05:05.974872 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:05.974836 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db"] Oct 30 01:05:05.975046 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:05.974994 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db" Oct 30 01:05:06.000040 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:05.992092 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-69dc90\"" Oct 30 01:05:06.000040 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:05.992234 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-69dc90-dockercfg-n8hx9\"" Oct 30 01:05:06.065856 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:06.065819 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg8z7\" (UniqueName: \"kubernetes.io/projected/53efc96a-851b-49fd-afee-b158c3a849b6-kube-api-access-cg8z7\") pod \"payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db\" (UID: \"53efc96a-851b-49fd-afee-b158c3a849b6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db" Oct 30 01:05:06.066008 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:06.065965 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53efc96a-851b-49fd-afee-b158c3a849b6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db\" (UID: \"53efc96a-851b-49fd-afee-b158c3a849b6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db" Oct 30 01:05:06.167803 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:06.167751 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg8z7\" (UniqueName: \"kubernetes.io/projected/53efc96a-851b-49fd-afee-b158c3a849b6-kube-api-access-cg8z7\") pod \"payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db\" (UID: \"53efc96a-851b-49fd-afee-b158c3a849b6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db" Oct 30 01:05:06.167972 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:06.167895 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53efc96a-851b-49fd-afee-b158c3a849b6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db\" (UID: \"53efc96a-851b-49fd-afee-b158c3a849b6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db" Oct 30 01:05:06.168707 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:06.168685 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53efc96a-851b-49fd-afee-b158c3a849b6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db\" (UID: \"53efc96a-851b-49fd-afee-b158c3a849b6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db" Oct 30 01:05:06.188898 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:06.188822 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg8z7\" (UniqueName: \"kubernetes.io/projected/53efc96a-851b-49fd-afee-b158c3a849b6-kube-api-access-cg8z7\") pod \"payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db\" (UID: \"53efc96a-851b-49fd-afee-b158c3a849b6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db" Oct 30 01:05:06.293694 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:06.293662 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db" Oct 30 01:05:06.637542 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:06.637512 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db"] Oct 30 01:05:06.639706 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:05:06.639676 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53efc96a_851b_49fd_afee_b158c3a849b6.slice/crio-94c26529e2fd1fd4bd78707a83f75f11a4ad8511edfca1e53830c17118461de2 WatchSource:0}: Error finding container 94c26529e2fd1fd4bd78707a83f75f11a4ad8511edfca1e53830c17118461de2: Status 404 returned error can't find the container with id 94c26529e2fd1fd4bd78707a83f75f11a4ad8511edfca1e53830c17118461de2 Oct 30 01:05:06.870539 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:06.870505 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db" event={"ID":"53efc96a-851b-49fd-afee-b158c3a849b6","Type":"ContainerStarted","Data":"d19cc08c3def581d9ef767b88d9ac1ca660b4788afd09ead4d0d5f5573d25eea"} Oct 30 01:05:06.870720 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:06.870558 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db" event={"ID":"53efc96a-851b-49fd-afee-b158c3a849b6","Type":"ContainerStarted","Data":"94c26529e2fd1fd4bd78707a83f75f11a4ad8511edfca1e53830c17118461de2"} Oct 30 01:05:06.870720 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:06.870586 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db" Oct 30 01:05:06.913542 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:06.913438 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db" podStartSLOduration=1.913424811 podStartE2EDuration="1.913424811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:06.911219551 +0000 UTC m=+1307.227087878" watchObservedRunningTime="2026-10-30 01:05:06.913424811 +0000 UTC m=+1307.229293136" Oct 30 01:05:07.338846 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:07.338807 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:08.217073 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:08.217039 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:08.391138 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:08.391099 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:09.038248 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:09.038199 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:10.761787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:10.761729 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:11.875972 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:11.875940 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8" Oct 30 01:05:12.370110 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:12.370070 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:14.245076 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:14.245044 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:14.861165 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:14.861132 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f" Oct 30 01:05:14.862024 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:14.861986 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257" Oct 30 01:05:15.901805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:15.901745 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:16.803360 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:16.803327 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:17.678068 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:17.678026 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:17.882223 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:17.882197 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db" Oct 30 01:05:19.349972 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:19.349935 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:21.001618 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:21.001585 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:21.593858 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:05:21.593818 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:05:22.731711 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:22.731676 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r"] Oct 30 01:05:22.737448 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:22.737425 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r" Oct 30 01:05:22.746609 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:22.746577 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-maas-api-tls\") pod \"maas-api-e2e-api-a-17d855-687c555c94-56z5r\" (UID: \"a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r" Oct 30 01:05:22.746731 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:22.746654 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-openshift-service-ca\") pod \"maas-api-e2e-api-a-17d855-687c555c94-56z5r\" (UID: \"a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r" Oct 30 01:05:22.746805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:22.746755 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78lbh\" (UniqueName: \"kubernetes.io/projected/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-kube-api-access-78lbh\") pod \"maas-api-e2e-api-a-17d855-687c555c94-56z5r\" (UID: \"a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r" Oct 30 01:05:22.746986 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:22.746967 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-metrics-certs\") pod \"maas-api-e2e-api-a-17d855-687c555c94-56z5r\" (UID: \"a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r" Oct 30 01:05:22.757046 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:22.757014 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r"] Oct 30 01:05:22.766676 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:22.766646 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-17d855-serving-cert\"" Oct 30 01:05:22.801889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:22.801859 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:22.848557 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:22.848522 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-metrics-certs\") pod \"maas-api-e2e-api-a-17d855-687c555c94-56z5r\" (UID: \"a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r" Oct 30 01:05:22.848742 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:22.848647 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-maas-api-tls\") pod \"maas-api-e2e-api-a-17d855-687c555c94-56z5r\" (UID: \"a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r" Oct 30 01:05:22.848742 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:22.848685 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-openshift-service-ca\") pod \"maas-api-e2e-api-a-17d855-687c555c94-56z5r\" (UID: \"a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r" Oct 30 01:05:22.848742 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:22.848722 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78lbh\" (UniqueName: \"kubernetes.io/projected/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-kube-api-access-78lbh\") pod \"maas-api-e2e-api-a-17d855-687c555c94-56z5r\" (UID: \"a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r" Oct 30 01:05:22.848958 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:05:22.848819 5300 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-serving-cert: secret "maas-api-e2e-api-a-17d855-serving-cert" not found Oct 30 01:05:22.848958 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:05:22.848895 5300 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-maas-api-tls podName:a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4 nodeName:}" failed. No retries permitted until 2026-10-30 01:05:23.348873527 +0000 UTC m=+1323.664741846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-maas-api-tls") pod "maas-api-e2e-api-a-17d855-687c555c94-56z5r" (UID: "a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4") : secret "maas-api-e2e-api-a-17d855-serving-cert" not found Oct 30 01:05:22.849383 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:22.849361 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-openshift-service-ca\") pod \"maas-api-e2e-api-a-17d855-687c555c94-56z5r\" (UID: \"a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r" Oct 30 01:05:22.851114 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:22.851093 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-metrics-certs\") pod \"maas-api-e2e-api-a-17d855-687c555c94-56z5r\" (UID: \"a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r" Oct 30 01:05:22.870221 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:22.870192 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78lbh\" (UniqueName: \"kubernetes.io/projected/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-kube-api-access-78lbh\") pod \"maas-api-e2e-api-a-17d855-687c555c94-56z5r\" (UID: \"a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r" Oct 30 01:05:22.880247 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:22.880205 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:22.992415 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:22.992324 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:23.170700 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:23.170660 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:23.356732 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:23.356637 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-maas-api-tls\") pod \"maas-api-e2e-api-a-17d855-687c555c94-56z5r\" (UID: \"a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r" Oct 30 01:05:23.358995 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:23.358970 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-maas-api-tls\") pod \"maas-api-e2e-api-a-17d855-687c555c94-56z5r\" (UID: \"a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r" Oct 30 01:05:23.652184 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:23.652082 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r" Oct 30 01:05:23.965402 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:23.965367 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r"] Oct 30 01:05:23.965402 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:05:23.965384 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f0c6c2_021f_4b5b_9f4f_5d1f64bd76d4.slice/crio-182de75c27e0571afe461b6fa1d3a494c59ae56f1a5251577a20b5bad3b01c71 WatchSource:0}: Error finding container 182de75c27e0571afe461b6fa1d3a494c59ae56f1a5251577a20b5bad3b01c71: Status 404 returned error can't find the container with id 182de75c27e0571afe461b6fa1d3a494c59ae56f1a5251577a20b5bad3b01c71 Oct 30 01:05:23.997178 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:23.997142 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r" event={"ID":"a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4","Type":"ContainerStarted","Data":"182de75c27e0571afe461b6fa1d3a494c59ae56f1a5251577a20b5bad3b01c71"} Oct 30 01:05:24.606506 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:24.606463 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:24.706881 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:24.706755 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:24.823670 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:24.823631 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:25.005980 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:25.005927 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r" event={"ID":"a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4","Type":"ContainerStarted","Data":"0c9d268d6aaa208c8e34b4d6ace6e360c46c36e44bd363acf70a7c85d30e7d1b"} Oct 30 01:05:25.006381 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:25.006036 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r" Oct 30 01:05:25.053450 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:25.053394 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r" podStartSLOduration=2.635019191 podStartE2EDuration="3.053380204s" totalImagesPullingTime="418.361013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:25.051225353 +0000 UTC m=+1325.367093677" watchObservedRunningTime="2026-10-30 01:05:25.053380204 +0000 UTC m=+1325.369248527" Oct 30 01:05:26.343873 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:26.343831 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:28.244388 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:28.244331 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:29.727240 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:29.727207 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:31.021260 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:31.021228 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r" Oct 30 01:05:31.382169 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:31.382072 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:31.758958 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:31.758922 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:35.524302 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:35.524267 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:35.625961 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:35.625902 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:36.080011 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:36.079969 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:36.538815 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:36.538777 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:37.270827 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:37.270778 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:37.808182 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:37.808140 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:37.934194 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:37.934155 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:38.147994 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:38.147895 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:40.006093 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:40.006057 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:41.920070 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:41.920032 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:42.064145 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:42.064107 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:42.184227 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:42.184125 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:43.767273 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:43.767239 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:43.982095 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:43.982057 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:44.080964 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:44.080885 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:45.541045 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:45.541011 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:47.264946 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:47.264915 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:48.950866 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:48.950825 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:50.567709 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:50.567676 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:51.558209 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:51.558165 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:51.605026 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:05:51.604977 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:05:52.204580 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:52.204496 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:52.450492 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:52.450458 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:53.382922 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:53.382878 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:53.741266 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:53.741233 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:54.561324 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:54.561287 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:56.334702 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:56.334663 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:56.713016 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:56.712908 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:56.916088 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:56.916045 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:57.028209 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:57.028171 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:58.271894 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:58.271860 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:58.787926 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:58.787894 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:58.886093 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:58.886052 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:05:59.074108 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:59.074013 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn"] Oct 30 01:05:59.080290 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:59.080258 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn" Oct 30 01:05:59.118043 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:59.118016 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-288da6\"" Oct 30 01:05:59.118259 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:59.118041 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-288da6-dockercfg-ndb5d\"" Oct 30 01:05:59.130256 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:59.130218 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn"] Oct 30 01:05:59.264771 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:59.264736 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c9209c7-2686-4759-96f2-efb5092b4536-plugins-config-volume\") pod \"payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn\" (UID: \"8c9209c7-2686-4759-96f2-efb5092b4536\") " pod="openshift-ingress/payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn" Oct 30 01:05:59.264961 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:59.264938 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdwxl\" (UniqueName: \"kubernetes.io/projected/8c9209c7-2686-4759-96f2-efb5092b4536-kube-api-access-gdwxl\") pod \"payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn\" (UID: \"8c9209c7-2686-4759-96f2-efb5092b4536\") " pod="openshift-ingress/payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn" Oct 30 01:05:59.366145 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:59.366059 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdwxl\" (UniqueName: \"kubernetes.io/projected/8c9209c7-2686-4759-96f2-efb5092b4536-kube-api-access-gdwxl\") pod \"payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn\" (UID: \"8c9209c7-2686-4759-96f2-efb5092b4536\") " pod="openshift-ingress/payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn" Oct 30 01:05:59.366544 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:59.366219 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c9209c7-2686-4759-96f2-efb5092b4536-plugins-config-volume\") pod \"payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn\" (UID: \"8c9209c7-2686-4759-96f2-efb5092b4536\") " pod="openshift-ingress/payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn" Oct 30 01:05:59.367477 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:59.367452 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c9209c7-2686-4759-96f2-efb5092b4536-plugins-config-volume\") pod \"payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn\" (UID: \"8c9209c7-2686-4759-96f2-efb5092b4536\") " pod="openshift-ingress/payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn" Oct 30 01:05:59.403470 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:59.403436 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdwxl\" (UniqueName: \"kubernetes.io/projected/8c9209c7-2686-4759-96f2-efb5092b4536-kube-api-access-gdwxl\") pod \"payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn\" (UID: \"8c9209c7-2686-4759-96f2-efb5092b4536\") " pod="openshift-ingress/payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn" Oct 30 01:05:59.423331 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:59.423297 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn" Oct 30 01:05:59.739543 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:05:59.739514 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn"] Oct 30 01:05:59.740024 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:05:59.740000 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c9209c7_2686_4759_96f2_efb5092b4536.slice/crio-6641fa912137cc2aada56552dc6d187a10c363db829471b73aa13071b5acb4e0 WatchSource:0}: Error finding container 6641fa912137cc2aada56552dc6d187a10c363db829471b73aa13071b5acb4e0: Status 404 returned error can't find the container with id 6641fa912137cc2aada56552dc6d187a10c363db829471b73aa13071b5acb4e0 Oct 30 01:06:00.085253 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:00.085167 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:00.359463 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:00.359367 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn" event={"ID":"8c9209c7-2686-4759-96f2-efb5092b4536","Type":"ContainerStarted","Data":"57ab1ec66ceee954d9db0a1c4c1d780ab5d6bf5bd254760438dae80c79225d7a"} Oct 30 01:06:00.359463 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:00.359459 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn" Oct 30 01:06:00.359663 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:00.359480 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn" event={"ID":"8c9209c7-2686-4759-96f2-efb5092b4536","Type":"ContainerStarted","Data":"6641fa912137cc2aada56552dc6d187a10c363db829471b73aa13071b5acb4e0"} Oct 30 01:06:00.405537 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:00.405468 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn" podStartSLOduration=1.405450049 podStartE2EDuration="1.405450049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:00.403587727 +0000 UTC m=+1360.719456053" watchObservedRunningTime="2026-10-30 01:06:00.405450049 +0000 UTC m=+1360.721318387" Oct 30 01:06:01.853078 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:01.852995 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:03.417139 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:03.417105 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7"] Oct 30 01:06:03.545292 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:03.545260 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:03.632557 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:03.632520 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:05.237685 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:05.237656 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:06.953256 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:06.953216 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:08.194792 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:08.194721 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:08.417718 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:08.417680 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7" podUID="abb97f33-12e0-4e9c-8e7c-842363ff915e" containerName="payload-processing" containerID="cri-o://8c77b00e86748a3c81eac2c5c7548effc58cdf5f4872a515b16e7b60a3ae0229" gracePeriod=25 Oct 30 01:06:08.534624 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:08.534585 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66"] Oct 30 01:06:08.539624 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:08.539599 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66" Oct 30 01:06:08.558573 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:08.558539 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66"] Oct 30 01:06:08.569314 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:08.569266 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-260268-serving-cert\"" Oct 30 01:06:08.609118 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:08.609080 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:08.617579 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:08.617543 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e0387dc-9b10-4e10-b892-a027d042f83f-metrics-certs\") pod \"maas-api-e2e-api-a-260268-7b898b5f48-clq66\" (UID: \"1e0387dc-9b10-4e10-b892-a027d042f83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66" Oct 30 01:06:08.617794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:08.617696 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs6h8\" (UniqueName: \"kubernetes.io/projected/1e0387dc-9b10-4e10-b892-a027d042f83f-kube-api-access-cs6h8\") pod \"maas-api-e2e-api-a-260268-7b898b5f48-clq66\" (UID: \"1e0387dc-9b10-4e10-b892-a027d042f83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66" Oct 30 01:06:08.617794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:08.617742 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e0387dc-9b10-4e10-b892-a027d042f83f-maas-api-tls\") pod \"maas-api-e2e-api-a-260268-7b898b5f48-clq66\" (UID: \"1e0387dc-9b10-4e10-b892-a027d042f83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66" Oct 30 01:06:08.617959 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:08.617889 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0387dc-9b10-4e10-b892-a027d042f83f-openshift-service-ca\") pod \"maas-api-e2e-api-a-260268-7b898b5f48-clq66\" (UID: \"1e0387dc-9b10-4e10-b892-a027d042f83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66" Oct 30 01:06:08.689948 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:08.689901 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:08.719505 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:08.719467 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e0387dc-9b10-4e10-b892-a027d042f83f-maas-api-tls\") pod \"maas-api-e2e-api-a-260268-7b898b5f48-clq66\" (UID: \"1e0387dc-9b10-4e10-b892-a027d042f83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66" Oct 30 01:06:08.719713 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:08.719629 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0387dc-9b10-4e10-b892-a027d042f83f-openshift-service-ca\") pod \"maas-api-e2e-api-a-260268-7b898b5f48-clq66\" (UID: \"1e0387dc-9b10-4e10-b892-a027d042f83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66" Oct 30 01:06:08.719822 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:08.719792 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e0387dc-9b10-4e10-b892-a027d042f83f-metrics-certs\") pod \"maas-api-e2e-api-a-260268-7b898b5f48-clq66\" (UID: \"1e0387dc-9b10-4e10-b892-a027d042f83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66" Oct 30 01:06:08.719936 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:08.719921 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs6h8\" (UniqueName: \"kubernetes.io/projected/1e0387dc-9b10-4e10-b892-a027d042f83f-kube-api-access-cs6h8\") pod \"maas-api-e2e-api-a-260268-7b898b5f48-clq66\" (UID: \"1e0387dc-9b10-4e10-b892-a027d042f83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66" Oct 30 01:06:08.720442 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:08.720421 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0387dc-9b10-4e10-b892-a027d042f83f-openshift-service-ca\") pod \"maas-api-e2e-api-a-260268-7b898b5f48-clq66\" (UID: \"1e0387dc-9b10-4e10-b892-a027d042f83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66" Oct 30 01:06:08.722089 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:08.722069 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e0387dc-9b10-4e10-b892-a027d042f83f-maas-api-tls\") pod \"maas-api-e2e-api-a-260268-7b898b5f48-clq66\" (UID: \"1e0387dc-9b10-4e10-b892-a027d042f83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66" Oct 30 01:06:08.722228 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:08.722209 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e0387dc-9b10-4e10-b892-a027d042f83f-metrics-certs\") pod \"maas-api-e2e-api-a-260268-7b898b5f48-clq66\" (UID: \"1e0387dc-9b10-4e10-b892-a027d042f83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66" Oct 30 01:06:08.741336 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:08.741309 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs6h8\" (UniqueName: \"kubernetes.io/projected/1e0387dc-9b10-4e10-b892-a027d042f83f-kube-api-access-cs6h8\") pod \"maas-api-e2e-api-a-260268-7b898b5f48-clq66\" (UID: \"1e0387dc-9b10-4e10-b892-a027d042f83f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66" Oct 30 01:06:08.790170 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:08.790082 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:08.853908 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:08.853870 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66" Oct 30 01:06:09.188447 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:06:09.188414 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e0387dc_9b10_4e10_b892_a027d042f83f.slice/crio-18f08c25a027fc22a1898886dc4b825b5ffe7b248515306b8f6517db62665524 WatchSource:0}: Error finding container 18f08c25a027fc22a1898886dc4b825b5ffe7b248515306b8f6517db62665524: Status 404 returned error can't find the container with id 18f08c25a027fc22a1898886dc4b825b5ffe7b248515306b8f6517db62665524 Oct 30 01:06:09.188604 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:09.188509 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66"] Oct 30 01:06:09.190066 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:09.190047 5300 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:06:09.432044 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:09.432002 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66" event={"ID":"1e0387dc-9b10-4e10-b892-a027d042f83f","Type":"ContainerStarted","Data":"18f08c25a027fc22a1898886dc4b825b5ffe7b248515306b8f6517db62665524"} Oct 30 01:06:09.708520 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:09.708482 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:10.190859 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:10.190822 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:10.291011 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:10.290980 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:10.406377 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:10.406306 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:10.444843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:10.444711 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66" event={"ID":"1e0387dc-9b10-4e10-b892-a027d042f83f","Type":"ContainerStarted","Data":"6716191b1a8a553e435ca9a79fd7f6dcc41f0b30ca83a5799fe4c2a223ccf0b7"} Oct 30 01:06:10.445798 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:10.445738 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66" Oct 30 01:06:10.492284 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:10.492143 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66" podStartSLOduration=2.115616957 podStartE2EDuration="2.492120986s" totalImagesPullingTime="376.504029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:10.488383057 +0000 UTC m=+1370.804251384" watchObservedRunningTime="2026-10-30 01:06:10.492120986 +0000 UTC m=+1370.807989311" Oct 30 01:06:11.368973 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:11.368942 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn" Oct 30 01:06:12.169178 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:12.169141 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:13.886610 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:13.886575 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:14.740796 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:14.740742 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:14.843130 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:14.843099 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:15.949213 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:15.949182 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:16.467394 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:16.467362 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:16.563700 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:16.563669 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:17.028030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:17.027995 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7" podUID="abb97f33-12e0-4e9c-8e7c-842363ff915e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:06:17.466699 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:17.466622 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66" Oct 30 01:06:17.821686 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:17.821651 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:19.533405 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:19.533375 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:21.251093 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:21.251056 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:21.594416 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:06:21.594330 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:06:22.916588 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:22.916542 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:24.496196 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:24.496157 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:24.590391 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:24.590360 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:24.691486 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:24.691438 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:26.448662 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:26.448627 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:27.028558 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:27.028517 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7" podUID="abb97f33-12e0-4e9c-8e7c-842363ff915e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:06:28.224784 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:28.219401 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:28.589963 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:28.589924 5300 generic.go:356] "Generic (PLEG): container finished" podID="abb97f33-12e0-4e9c-8e7c-842363ff915e" containerID="8c77b00e86748a3c81eac2c5c7548effc58cdf5f4872a515b16e7b60a3ae0229" exitCode=0 Oct 30 01:06:28.590124 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:28.589993 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7" event={"ID":"abb97f33-12e0-4e9c-8e7c-842363ff915e","Type":"ContainerDied","Data":"8c77b00e86748a3c81eac2c5c7548effc58cdf5f4872a515b16e7b60a3ae0229"} Oct 30 01:06:28.734918 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:28.734893 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7" Oct 30 01:06:28.785976 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:28.785933 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abb97f33-12e0-4e9c-8e7c-842363ff915e-kube-api-access-rf5gg\" (UniqueName: \"kubernetes.io/projected/abb97f33-12e0-4e9c-8e7c-842363ff915e-kube-api-access-rf5gg\") pod \"abb97f33-12e0-4e9c-8e7c-842363ff915e\" (UID: \"abb97f33-12e0-4e9c-8e7c-842363ff915e\") " Oct 30 01:06:28.786188 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:28.786171 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abb97f33-12e0-4e9c-8e7c-842363ff915e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb97f33-12e0-4e9c-8e7c-842363ff915e-plugins-config-volume\") pod \"abb97f33-12e0-4e9c-8e7c-842363ff915e\" (UID: \"abb97f33-12e0-4e9c-8e7c-842363ff915e\") " Oct 30 01:06:28.786607 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:28.786579 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abb97f33-12e0-4e9c-8e7c-842363ff915e-plugins-config-volume" pod "abb97f33-12e0-4e9c-8e7c-842363ff915e" (UID: "abb97f33-12e0-4e9c-8e7c-842363ff915e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:06:28.786856 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:28.786836 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb97f33-12e0-4e9c-8e7c-842363ff915e-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:06:28.788322 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:28.788298 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb97f33-12e0-4e9c-8e7c-842363ff915e-kube-api-access-rf5gg" pod "abb97f33-12e0-4e9c-8e7c-842363ff915e" (UID: "abb97f33-12e0-4e9c-8e7c-842363ff915e"). InnerVolumeSpecName "kube-api-access-rf5gg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:06:28.889382 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:28.889336 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf5gg\" (UniqueName: \"kubernetes.io/projected/abb97f33-12e0-4e9c-8e7c-842363ff915e-kube-api-access-rf5gg\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:06:29.106903 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:29.089095 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:29.229544 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:29.229424 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:29.624115 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:29.624076 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7" event={"ID":"abb97f33-12e0-4e9c-8e7c-842363ff915e","Type":"ContainerDied","Data":"b13e3957ef9f4b8fdf2c9e6dd716f593ef42c8b695ec7e830b098fc01e6a415f"} Oct 30 01:06:29.624413 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:29.624394 5300 scope.go:122] "RemoveContainer" containerID="8c77b00e86748a3c81eac2c5c7548effc58cdf5f4872a515b16e7b60a3ae0229" Oct 30 01:06:29.624709 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:29.624691 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7" Oct 30 01:06:29.795863 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:29.795817 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7"] Oct 30 01:06:29.801233 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:29.801190 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb3370cb-5cd6d77f9b-m77c7"] Oct 30 01:06:30.215301 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:30.215267 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abb97f33-12e0-4e9c-8e7c-842363ff915e" path="/var/lib/kubelet/pods/abb97f33-12e0-4e9c-8e7c-842363ff915e/volumes" Oct 30 01:06:30.239892 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:30.232018 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:32.155893 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:32.155852 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:32.260079 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:32.260047 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:33.979223 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:33.979180 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:35.654926 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:35.654889 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:37.528187 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:37.528141 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:39.273228 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:39.273181 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:41.156563 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:41.156524 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:42.745593 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:42.745555 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:42.830201 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:42.830163 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:42.934448 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:42.934411 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:43.189815 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:43.189699 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:43.825565 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:43.825537 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:44.389482 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:44.389437 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:44.656997 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:44.656907 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:45.197169 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:45.197134 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:46.326050 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:46.326015 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:47.217376 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.217342 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:47.648498 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.648461 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw"] Oct 30 01:06:47.649230 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.649213 5300 cpu_manager.go:414] "containerMap: removing container" podUID="abb97f33-12e0-4e9c-8e7c-842363ff915e" containerName="payload-processing" Oct 30 01:06:47.649281 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.649231 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abb97f33-12e0-4e9c-8e7c-842363ff915e" containerName="payload-processing" Oct 30 01:06:47.649388 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.649370 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="abb97f33-12e0-4e9c-8e7c-842363ff915e" containerName="payload-processing" Oct 30 01:06:47.653483 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.653463 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.674595 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.674567 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-13a3e2-openshift-default-dockercfg-q67kg\"" Oct 30 01:06:47.690113 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.690084 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw"] Oct 30 01:06:47.696941 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.696912 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:47.778536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.778493 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istiod-ca-cert\") pod \"e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.778741 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.778643 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th2bz\" (UniqueName: \"kubernetes.io/projected/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-kube-api-access-th2bz\") pod \"e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.778844 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.778808 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-credential-socket\") pod \"e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.778907 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.778850 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-workload-socket\") pod \"e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.778907 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.778898 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-token\") pod \"e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.779123 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.779089 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-data\") pod \"e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.779262 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.779161 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-podinfo\") pod \"e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.779262 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.779224 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-workload-certs\") pod \"e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.779390 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.779353 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-envoy\") pod \"e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.881683 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.881647 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-data\") pod \"e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.881907 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.881689 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-podinfo\") pod \"e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.881907 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.881745 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-workload-certs\") pod \"e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.882240 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.882218 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-data\") pod \"e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.882713 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.882686 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-envoy\") pod \"e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.882894 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.882807 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istiod-ca-cert\") pod \"e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.882894 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.882875 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th2bz\" (UniqueName: \"kubernetes.io/projected/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-kube-api-access-th2bz\") pod \"e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.883032 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.882970 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-credential-socket\") pod \"e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.883032 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.883005 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-workload-socket\") pod \"e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.883134 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.883055 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-token\") pod \"e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.883390 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.883370 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-workload-certs\") pod \"e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.884149 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.884124 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-credential-socket\") pod \"e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.884849 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.884821 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istiod-ca-cert\") pod \"e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.885125 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.885100 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-workload-socket\") pod \"e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.900604 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.899317 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-podinfo\") pod \"e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.901420 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.901395 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-envoy\") pod \"e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.909846 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.909801 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-token\") pod \"e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.912846 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.912810 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th2bz\" (UniqueName: \"kubernetes.io/projected/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-kube-api-access-th2bz\") pod \"e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:47.970699 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:47.970671 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:48.292414 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:48.292379 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:48.322727 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:48.307843 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw"] Oct 30 01:06:48.332579 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:48.332533 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:06:48.332713 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:48.332627 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:06:48.332713 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:48.332671 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:06:48.387968 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:48.387938 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:49.073667 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:49.073628 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" event={"ID":"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2","Type":"ContainerStarted","Data":"f916fe3c48aaad148fdacf849fa41dc4e50ea86dc4e2d80e52712f4f047d6ce4"} Oct 30 01:06:49.074096 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:49.073681 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" event={"ID":"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2","Type":"ContainerStarted","Data":"d553131254a01a98a31e57bd99fca0d583dffadf07e025436492b12a7b286076"} Oct 30 01:06:49.145780 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:49.145714 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" podStartSLOduration=2.145700907 podStartE2EDuration="2.145700907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:49.131956058 +0000 UTC m=+1409.447824386" watchObservedRunningTime="2026-10-30 01:06:49.145700907 +0000 UTC m=+1409.461569231" Oct 30 01:06:49.606025 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:49.605985 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:49.706123 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:49.706092 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:49.971282 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:49.971254 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:49.977220 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:49.977194 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:50.080281 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:50.080250 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:50.081332 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:50.081300 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:06:50.210569 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:50.210531 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:51.552495 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:51.552459 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:51.606239 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:06:51.606199 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:06:51.657284 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:51.657246 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:51.916659 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:51.916581 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:52.019522 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:52.019477 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:52.273577 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:52.273525 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:54.121191 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:54.121141 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:54.219370 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:54.219339 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:54.317235 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:54.317202 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg"] Oct 30 01:06:54.321988 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:54.321964 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg" Oct 30 01:06:54.339319 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:54.339292 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg"] Oct 30 01:06:54.347906 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:54.347881 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-13a3e2\"" Oct 30 01:06:54.348062 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:54.348042 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-13a3e2-dockercfg-vzpmp\"" Oct 30 01:06:54.405797 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:54.405694 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq77s\" (UniqueName: \"kubernetes.io/projected/58101d14-ecc3-456c-bb6a-e1472585152c-kube-api-access-bq77s\") pod \"payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg\" (UID: \"58101d14-ecc3-456c-bb6a-e1472585152c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg" Oct 30 01:06:54.406099 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:54.406073 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58101d14-ecc3-456c-bb6a-e1472585152c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg\" (UID: \"58101d14-ecc3-456c-bb6a-e1472585152c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg" Oct 30 01:06:54.495786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:54.492817 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw"] Oct 30 01:06:54.516785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:54.498787 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw" Oct 30 01:06:54.516785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:54.505632 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:54.516785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:54.506743 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw"] Oct 30 01:06:54.516785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:54.507616 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94t9f\" (UniqueName: \"kubernetes.io/projected/b80a1557-9093-451b-aa8d-f088de9b2d61-kube-api-access-94t9f\") pod \"payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw\" (UID: \"b80a1557-9093-451b-aa8d-f088de9b2d61\") " pod="openshift-ingress/payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw" Oct 30 01:06:54.516785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:54.507668 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq77s\" (UniqueName: \"kubernetes.io/projected/58101d14-ecc3-456c-bb6a-e1472585152c-kube-api-access-bq77s\") pod \"payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg\" (UID: \"58101d14-ecc3-456c-bb6a-e1472585152c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg" Oct 30 01:06:54.516785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:54.507753 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58101d14-ecc3-456c-bb6a-e1472585152c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg\" (UID: \"58101d14-ecc3-456c-bb6a-e1472585152c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg" Oct 30 01:06:54.516785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:54.507888 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b80a1557-9093-451b-aa8d-f088de9b2d61-plugins-config-volume\") pod \"payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw\" (UID: \"b80a1557-9093-451b-aa8d-f088de9b2d61\") " pod="openshift-ingress/payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw" Oct 30 01:06:54.516785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:54.508988 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58101d14-ecc3-456c-bb6a-e1472585152c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg\" (UID: \"58101d14-ecc3-456c-bb6a-e1472585152c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg" Oct 30 01:06:54.564485 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:54.564449 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq77s\" (UniqueName: \"kubernetes.io/projected/58101d14-ecc3-456c-bb6a-e1472585152c-kube-api-access-bq77s\") pod \"payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg\" (UID: \"58101d14-ecc3-456c-bb6a-e1472585152c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg" Oct 30 01:06:54.609004 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:54.608967 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94t9f\" (UniqueName: \"kubernetes.io/projected/b80a1557-9093-451b-aa8d-f088de9b2d61-kube-api-access-94t9f\") pod \"payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw\" (UID: \"b80a1557-9093-451b-aa8d-f088de9b2d61\") " pod="openshift-ingress/payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw" Oct 30 01:06:54.609398 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:54.609371 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b80a1557-9093-451b-aa8d-f088de9b2d61-plugins-config-volume\") pod \"payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw\" (UID: \"b80a1557-9093-451b-aa8d-f088de9b2d61\") " pod="openshift-ingress/payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw" Oct 30 01:06:54.611074 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:54.611038 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b80a1557-9093-451b-aa8d-f088de9b2d61-plugins-config-volume\") pod \"payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw\" (UID: \"b80a1557-9093-451b-aa8d-f088de9b2d61\") " pod="openshift-ingress/payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw" Oct 30 01:06:54.635960 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:54.635879 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94t9f\" (UniqueName: \"kubernetes.io/projected/b80a1557-9093-451b-aa8d-f088de9b2d61-kube-api-access-94t9f\") pod \"payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw\" (UID: \"b80a1557-9093-451b-aa8d-f088de9b2d61\") " pod="openshift-ingress/payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw" Oct 30 01:06:54.639247 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:54.639217 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg" Oct 30 01:06:54.858040 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:54.858007 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw" Oct 30 01:06:54.976209 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:54.976171 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg"] Oct 30 01:06:54.987503 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:06:54.987470 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58101d14_ecc3_456c_bb6a_e1472585152c.slice/crio-faa3037970ebc4afc656cf1acbf494fee7a9d39980eba68830cfadb0362ae082 WatchSource:0}: Error finding container faa3037970ebc4afc656cf1acbf494fee7a9d39980eba68830cfadb0362ae082: Status 404 returned error can't find the container with id faa3037970ebc4afc656cf1acbf494fee7a9d39980eba68830cfadb0362ae082 Oct 30 01:06:55.124499 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:55.124460 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg" event={"ID":"58101d14-ecc3-456c-bb6a-e1472585152c","Type":"ContainerStarted","Data":"8f116f4ca1570d296f21f1850ccea1a35d20e3cead1bea6b7015c8f68707e7d6"} Oct 30 01:06:55.124917 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:55.124525 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg" event={"ID":"58101d14-ecc3-456c-bb6a-e1472585152c","Type":"ContainerStarted","Data":"faa3037970ebc4afc656cf1acbf494fee7a9d39980eba68830cfadb0362ae082"} Oct 30 01:06:55.124917 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:55.124557 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg" Oct 30 01:06:55.171001 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:55.170954 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg" podStartSLOduration=1.170939105 podStartE2EDuration="1.170939105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:55.166958519 +0000 UTC m=+1415.482826844" watchObservedRunningTime="2026-10-30 01:06:55.170939105 +0000 UTC m=+1415.486807454" Oct 30 01:06:55.183993 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:06:55.183926 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb80a1557_9093_451b_aa8d_f088de9b2d61.slice/crio-452a15336579a0f60a51345aa7989f444937c6515133ac5d89a77042107a726c WatchSource:0}: Error finding container 452a15336579a0f60a51345aa7989f444937c6515133ac5d89a77042107a726c: Status 404 returned error can't find the container with id 452a15336579a0f60a51345aa7989f444937c6515133ac5d89a77042107a726c Oct 30 01:06:55.189545 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:55.186929 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw"] Oct 30 01:06:56.133000 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:56.132963 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw" event={"ID":"b80a1557-9093-451b-aa8d-f088de9b2d61","Type":"ContainerStarted","Data":"4d6a133638129c3ffc8c4ae69023104f0f31195abc47515c1fd09a8e12b0ae1d"} Oct 30 01:06:56.133460 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:56.133019 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw" event={"ID":"b80a1557-9093-451b-aa8d-f088de9b2d61","Type":"ContainerStarted","Data":"452a15336579a0f60a51345aa7989f444937c6515133ac5d89a77042107a726c"} Oct 30 01:06:56.133460 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:56.133245 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw" Oct 30 01:06:56.177046 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:56.177004 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw" podStartSLOduration=2.176983556 podStartE2EDuration="2.176983556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:56.174313405 +0000 UTC m=+1416.490181727" watchObservedRunningTime="2026-10-30 01:06:56.176983556 +0000 UTC m=+1416.492851880" Oct 30 01:06:56.462363 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:56.462277 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:58.297452 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:58.297408 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:06:59.842601 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:59.842568 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm"] Oct 30 01:06:59.843047 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:59.842748 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm" podUID="c817200c-b0bc-4fb9-a685-d2c7c08da0a5" containerName="maas-api" containerID="cri-o://84bd857d54ab5021f373d572b1e935c87934a5435e30accfb9cbb89477895b73" gracePeriod=30 Oct 30 01:06:59.897193 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:06:59.897147 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:00.130701 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:00.130677 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm" Oct 30 01:07:00.170979 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:00.170906 5300 generic.go:356] "Generic (PLEG): container finished" podID="c817200c-b0bc-4fb9-a685-d2c7c08da0a5" containerID="84bd857d54ab5021f373d572b1e935c87934a5435e30accfb9cbb89477895b73" exitCode=0 Oct 30 01:07:00.171224 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:00.171191 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm" Oct 30 01:07:00.171561 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:00.171532 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm" event={"ID":"c817200c-b0bc-4fb9-a685-d2c7c08da0a5","Type":"ContainerDied","Data":"84bd857d54ab5021f373d572b1e935c87934a5435e30accfb9cbb89477895b73"} Oct 30 01:07:00.175547 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:00.171599 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm" event={"ID":"c817200c-b0bc-4fb9-a685-d2c7c08da0a5","Type":"ContainerDied","Data":"0ef0955851923f4b6215172677611b7f01e7a257c035b2c36350d43786df9194"} Oct 30 01:07:00.175547 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:00.171646 5300 scope.go:122] "RemoveContainer" containerID="84bd857d54ab5021f373d572b1e935c87934a5435e30accfb9cbb89477895b73" Oct 30 01:07:00.226783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:00.203799 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-openshift-service-ca\") pod \"c817200c-b0bc-4fb9-a685-d2c7c08da0a5\" (UID: \"c817200c-b0bc-4fb9-a685-d2c7c08da0a5\") " Oct 30 01:07:00.226783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:00.203880 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-maas-api-tls\") pod \"c817200c-b0bc-4fb9-a685-d2c7c08da0a5\" (UID: \"c817200c-b0bc-4fb9-a685-d2c7c08da0a5\") " Oct 30 01:07:00.226783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:00.203958 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-metrics-certs\") pod \"c817200c-b0bc-4fb9-a685-d2c7c08da0a5\" (UID: \"c817200c-b0bc-4fb9-a685-d2c7c08da0a5\") " Oct 30 01:07:00.226783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:00.204182 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-kube-api-access-6km7n\" (UniqueName: \"kubernetes.io/projected/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-kube-api-access-6km7n\") pod \"c817200c-b0bc-4fb9-a685-d2c7c08da0a5\" (UID: \"c817200c-b0bc-4fb9-a685-d2c7c08da0a5\") " Oct 30 01:07:00.226783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:00.206164 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-openshift-service-ca" pod "c817200c-b0bc-4fb9-a685-d2c7c08da0a5" (UID: "c817200c-b0bc-4fb9-a685-d2c7c08da0a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:07:00.226783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:00.223318 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-kube-api-access-6km7n" pod "c817200c-b0bc-4fb9-a685-d2c7c08da0a5" (UID: "c817200c-b0bc-4fb9-a685-d2c7c08da0a5"). InnerVolumeSpecName "kube-api-access-6km7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:00.226783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:00.224004 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-metrics-certs" pod "c817200c-b0bc-4fb9-a685-d2c7c08da0a5" (UID: "c817200c-b0bc-4fb9-a685-d2c7c08da0a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:07:00.226783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:00.224103 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-maas-api-tls" pod "c817200c-b0bc-4fb9-a685-d2c7c08da0a5" (UID: "c817200c-b0bc-4fb9-a685-d2c7c08da0a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:07:00.274797 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:00.274156 5300 scope.go:122] "RemoveContainer" containerID="84bd857d54ab5021f373d572b1e935c87934a5435e30accfb9cbb89477895b73" Oct 30 01:07:00.274797 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:07:00.274612 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84bd857d54ab5021f373d572b1e935c87934a5435e30accfb9cbb89477895b73\": container with ID starting with 84bd857d54ab5021f373d572b1e935c87934a5435e30accfb9cbb89477895b73 not found: ID does not exist" containerID="84bd857d54ab5021f373d572b1e935c87934a5435e30accfb9cbb89477895b73" Oct 30 01:07:00.274797 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:00.274662 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84bd857d54ab5021f373d572b1e935c87934a5435e30accfb9cbb89477895b73"} err="failed to get container status \"84bd857d54ab5021f373d572b1e935c87934a5435e30accfb9cbb89477895b73\": rpc error: code = NotFound desc = could not find container \"84bd857d54ab5021f373d572b1e935c87934a5435e30accfb9cbb89477895b73\": container with ID starting with 84bd857d54ab5021f373d572b1e935c87934a5435e30accfb9cbb89477895b73 not found: ID does not exist" Oct 30 01:07:00.323202 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:00.322509 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-maas-api-tls\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:07:00.323202 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:00.322549 5300 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-metrics-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:07:00.323202 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:00.322564 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6km7n\" (UniqueName: \"kubernetes.io/projected/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-kube-api-access-6km7n\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:07:00.323202 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:00.322580 5300 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c817200c-b0bc-4fb9-a685-d2c7c08da0a5-openshift-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:07:00.535554 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:00.535517 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm"] Oct 30 01:07:00.551196 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:00.551160 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08065c8e-bf9f49d88-b5dpm"] Oct 30 01:07:00.662905 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:00.662774 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:00.752091 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:00.752001 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:00.849623 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:00.849455 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:01.275266 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:01.275237 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:01.822472 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:01.822422 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c817200c-b0bc-4fb9-a685-d2c7c08da0a5" path="/var/lib/kubelet/pods/c817200c-b0bc-4fb9-a685-d2c7c08da0a5/volumes" Oct 30 01:07:02.807011 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:02.806982 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:03.138332 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:03.138243 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:04.596977 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:04.596938 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:06.136442 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:06.136407 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg" Oct 30 01:07:06.384608 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:06.384575 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:07.076011 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:07.075980 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:07.144150 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:07.144122 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw" Oct 30 01:07:07.176839 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:07.176795 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:07.475134 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:07.475103 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:07.963730 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:07.963651 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx"] Oct 30 01:07:07.963979 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:07.963953 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx" podUID="f34da3d1-b871-4daa-b86a-5376585b1c1e" containerName="maas-api" containerID="cri-o://405ccbc40350d29f35e7911dc452f8a829207cfa7bd4390f128a36145ff706dc" gracePeriod=30 Oct 30 01:07:07.997112 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:07.996991 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:08.257189 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:08.257153 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:08.272973 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:08.272940 5300 generic.go:356] "Generic (PLEG): container finished" podID="f34da3d1-b871-4daa-b86a-5376585b1c1e" containerID="405ccbc40350d29f35e7911dc452f8a829207cfa7bd4390f128a36145ff706dc" exitCode=0 Oct 30 01:07:08.273115 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:08.273007 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx" event={"ID":"f34da3d1-b871-4daa-b86a-5376585b1c1e","Type":"ContainerDied","Data":"405ccbc40350d29f35e7911dc452f8a829207cfa7bd4390f128a36145ff706dc"} Oct 30 01:07:08.303300 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:08.303270 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx" Oct 30 01:07:08.398830 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:08.398791 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f34da3d1-b871-4daa-b86a-5376585b1c1e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f34da3d1-b871-4daa-b86a-5376585b1c1e-maas-api-tls\") pod \"f34da3d1-b871-4daa-b86a-5376585b1c1e\" (UID: \"f34da3d1-b871-4daa-b86a-5376585b1c1e\") " Oct 30 01:07:08.399038 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:08.398874 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f34da3d1-b871-4daa-b86a-5376585b1c1e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f34da3d1-b871-4daa-b86a-5376585b1c1e-openshift-service-ca\") pod \"f34da3d1-b871-4daa-b86a-5376585b1c1e\" (UID: \"f34da3d1-b871-4daa-b86a-5376585b1c1e\") " Oct 30 01:07:08.399119 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:08.399058 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f34da3d1-b871-4daa-b86a-5376585b1c1e-kube-api-access-rp74t\" (UniqueName: \"kubernetes.io/projected/f34da3d1-b871-4daa-b86a-5376585b1c1e-kube-api-access-rp74t\") pod \"f34da3d1-b871-4daa-b86a-5376585b1c1e\" (UID: \"f34da3d1-b871-4daa-b86a-5376585b1c1e\") " Oct 30 01:07:08.400502 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:08.399492 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f34da3d1-b871-4daa-b86a-5376585b1c1e-openshift-service-ca" pod "f34da3d1-b871-4daa-b86a-5376585b1c1e" (UID: "f34da3d1-b871-4daa-b86a-5376585b1c1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:07:08.400502 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:08.399645 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f34da3d1-b871-4daa-b86a-5376585b1c1e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f34da3d1-b871-4daa-b86a-5376585b1c1e-metrics-certs\") pod \"f34da3d1-b871-4daa-b86a-5376585b1c1e\" (UID: \"f34da3d1-b871-4daa-b86a-5376585b1c1e\") " Oct 30 01:07:08.400502 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:08.400321 5300 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f34da3d1-b871-4daa-b86a-5376585b1c1e-openshift-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:07:08.431725 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:08.431677 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f34da3d1-b871-4daa-b86a-5376585b1c1e-maas-api-tls" pod "f34da3d1-b871-4daa-b86a-5376585b1c1e" (UID: "f34da3d1-b871-4daa-b86a-5376585b1c1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:07:08.431950 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:08.431924 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f34da3d1-b871-4daa-b86a-5376585b1c1e-kube-api-access-rp74t" pod "f34da3d1-b871-4daa-b86a-5376585b1c1e" (UID: "f34da3d1-b871-4daa-b86a-5376585b1c1e"). InnerVolumeSpecName "kube-api-access-rp74t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:08.432171 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:08.432147 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f34da3d1-b871-4daa-b86a-5376585b1c1e-metrics-certs" pod "f34da3d1-b871-4daa-b86a-5376585b1c1e" (UID: "f34da3d1-b871-4daa-b86a-5376585b1c1e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:07:08.507641 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:08.507378 5300 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f34da3d1-b871-4daa-b86a-5376585b1c1e-metrics-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:07:08.507641 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:08.507409 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f34da3d1-b871-4daa-b86a-5376585b1c1e-maas-api-tls\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:07:08.507641 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:08.507422 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp74t\" (UniqueName: \"kubernetes.io/projected/f34da3d1-b871-4daa-b86a-5376585b1c1e-kube-api-access-rp74t\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:07:08.885165 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:08.885071 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:08.986180 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:08.986150 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:09.087915 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.087882 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:09.290272 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.290241 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx" Oct 30 01:07:09.290988 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.290263 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx" event={"ID":"f34da3d1-b871-4daa-b86a-5376585b1c1e","Type":"ContainerDied","Data":"04f13af169cf355bcbf6c594ab77f98e0aa74ca2221670b152cc3d277aa0ed3f"} Oct 30 01:07:09.291125 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.291046 5300 scope.go:122] "RemoveContainer" containerID="405ccbc40350d29f35e7911dc452f8a829207cfa7bd4390f128a36145ff706dc" Oct 30 01:07:09.362699 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.362667 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp"] Oct 30 01:07:09.363427 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.363411 5300 cpu_manager.go:414] "containerMap: removing container" podUID="f34da3d1-b871-4daa-b86a-5376585b1c1e" containerName="maas-api" Oct 30 01:07:09.363471 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.363429 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f34da3d1-b871-4daa-b86a-5376585b1c1e" containerName="maas-api" Oct 30 01:07:09.363471 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.363462 5300 cpu_manager.go:414] "containerMap: removing container" podUID="c817200c-b0bc-4fb9-a685-d2c7c08da0a5" containerName="maas-api" Oct 30 01:07:09.363471 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.363467 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c817200c-b0bc-4fb9-a685-d2c7c08da0a5" containerName="maas-api" Oct 30 01:07:09.363573 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.363564 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="f34da3d1-b871-4daa-b86a-5376585b1c1e" containerName="maas-api" Oct 30 01:07:09.363607 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.363579 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="c817200c-b0bc-4fb9-a685-d2c7c08da0a5" containerName="maas-api" Oct 30 01:07:09.368547 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.368526 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.388920 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.388889 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-262230-openshift-default-dockercfg-ks9dj\"" Oct 30 01:07:09.389275 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.389248 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp"] Oct 30 01:07:09.394660 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.394633 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx"] Oct 30 01:07:09.405802 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.403476 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb3370cb-7c4ff4d8df-47rzx"] Oct 30 01:07:09.421481 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.421454 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-istio-envoy\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.421640 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.421513 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d13883c2-ec67-4526-985d-f22c6673853f-istiod-ca-cert\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.421714 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.421689 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-workload-certs\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.421909 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.421877 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-credential-socket\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.422032 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.421929 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-istio-data\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.422084 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.422034 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d13883c2-ec67-4526-985d-f22c6673853f-istio-podinfo\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.422136 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.422112 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-workload-socket\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.422192 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.422176 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d13883c2-ec67-4526-985d-f22c6673853f-istio-token\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.422248 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.422217 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5shdr\" (UniqueName: \"kubernetes.io/projected/d13883c2-ec67-4526-985d-f22c6673853f-kube-api-access-5shdr\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.466105 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.466067 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:09.523601 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.523558 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d13883c2-ec67-4526-985d-f22c6673853f-istiod-ca-cert\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.523914 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.523706 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-workload-certs\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.523914 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.523856 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-credential-socket\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.523914 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.523888 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-istio-data\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.524056 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.523949 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d13883c2-ec67-4526-985d-f22c6673853f-istio-podinfo\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.524056 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.524009 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-workload-socket\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.524056 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.524033 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d13883c2-ec67-4526-985d-f22c6673853f-istio-token\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.524149 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.524066 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5shdr\" (UniqueName: \"kubernetes.io/projected/d13883c2-ec67-4526-985d-f22c6673853f-kube-api-access-5shdr\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.524149 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.524124 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-istio-envoy\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.524903 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.524875 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-istio-data\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.525371 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.525340 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d13883c2-ec67-4526-985d-f22c6673853f-istiod-ca-cert\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.525599 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.525578 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-workload-socket\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.525855 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.525829 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-workload-certs\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.526127 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.526104 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-credential-socket\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.527880 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.527832 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d13883c2-ec67-4526-985d-f22c6673853f-istio-podinfo\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.531061 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.531032 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-istio-envoy\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.548382 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.548308 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5shdr\" (UniqueName: \"kubernetes.io/projected/d13883c2-ec67-4526-985d-f22c6673853f-kube-api-access-5shdr\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.549420 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.549398 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d13883c2-ec67-4526-985d-f22c6673853f-istio-token\") pod \"e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.684538 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.684265 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:09.835048 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.834968 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f34da3d1-b871-4daa-b86a-5376585b1c1e" path="/var/lib/kubelet/pods/f34da3d1-b871-4daa-b86a-5376585b1c1e/volumes" Oct 30 01:07:09.958259 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:09.958227 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:10.022722 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:07:10.022687 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd13883c2_ec67_4526_985d_f22c6673853f.slice/crio-87af2982273ebb5abeffa76234d0645a2ef8142d19d39a28db5ca2f8430f0dfe WatchSource:0}: Error finding container 87af2982273ebb5abeffa76234d0645a2ef8142d19d39a28db5ca2f8430f0dfe: Status 404 returned error can't find the container with id 87af2982273ebb5abeffa76234d0645a2ef8142d19d39a28db5ca2f8430f0dfe Oct 30 01:07:10.023618 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:10.023574 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp"] Oct 30 01:07:10.026052 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:10.026015 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:07:10.026170 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:10.026094 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:07:10.026170 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:10.026148 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:07:10.308787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:10.305376 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" event={"ID":"d13883c2-ec67-4526-985d-f22c6673853f","Type":"ContainerStarted","Data":"1a8a522e7b395c6e2a170ad1cfee67b899241e218c1d080eae472f64f7b5527f"} Oct 30 01:07:10.308787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:10.305474 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" event={"ID":"d13883c2-ec67-4526-985d-f22c6673853f","Type":"ContainerStarted","Data":"87af2982273ebb5abeffa76234d0645a2ef8142d19d39a28db5ca2f8430f0dfe"} Oct 30 01:07:10.352818 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:10.352706 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" podStartSLOduration=1.352692621 podStartE2EDuration="1.352692621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:10.351839858 +0000 UTC m=+1430.667708184" watchObservedRunningTime="2026-10-30 01:07:10.352692621 +0000 UTC m=+1430.668560945" Oct 30 01:07:10.685214 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:10.685137 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:10.855561 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:10.855528 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:11.691373 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:11.691340 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:12.322028 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:12.321989 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:12.322960 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:12.322936 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:07:12.649137 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:12.649061 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:12.751087 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:12.751046 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:12.901264 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:12.901152 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:13.009356 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:13.009321 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:13.110082 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:13.110052 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:14.517584 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:14.517554 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:14.612705 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:14.612673 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:14.772489 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:14.772404 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:15.065139 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:15.065048 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f"] Oct 30 01:07:15.122468 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:15.122433 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257"] Oct 30 01:07:15.261008 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:15.260974 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:15.350316 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:15.350235 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:16.608696 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:16.608658 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:16.713370 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:16.713325 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:16.914393 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:16.914314 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:18.864109 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:18.864077 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:20.065852 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.065728 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f" podUID="a22c9a38-2f90-46f2-b924-0a28e85ac2fa" containerName="payload-processing" containerID="cri-o://951e706344460305c279f5b991194c967e6163fdea439c2c86266afb73496c7c" gracePeriod=25 Oct 30 01:07:20.123641 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.123599 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257" podUID="fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37" containerName="payload-pre-processing" containerID="cri-o://3301c565b1809fb85cad92634ee899d886520b2c2b8ceeb3bd2316e7f945111a" gracePeriod=25 Oct 30 01:07:20.411267 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.411236 5300 generic.go:356] "Generic (PLEG): container finished" podID="fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37" containerID="3301c565b1809fb85cad92634ee899d886520b2c2b8ceeb3bd2316e7f945111a" exitCode=0 Oct 30 01:07:20.411430 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.411364 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257" event={"ID":"fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37","Type":"ContainerDied","Data":"3301c565b1809fb85cad92634ee899d886520b2c2b8ceeb3bd2316e7f945111a"} Oct 30 01:07:20.413137 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.413112 5300 generic.go:356] "Generic (PLEG): container finished" podID="a22c9a38-2f90-46f2-b924-0a28e85ac2fa" containerID="951e706344460305c279f5b991194c967e6163fdea439c2c86266afb73496c7c" exitCode=0 Oct 30 01:07:20.413258 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.413159 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f" event={"ID":"a22c9a38-2f90-46f2-b924-0a28e85ac2fa","Type":"ContainerDied","Data":"951e706344460305c279f5b991194c967e6163fdea439c2c86266afb73496c7c"} Oct 30 01:07:20.413258 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.413201 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f" event={"ID":"a22c9a38-2f90-46f2-b924-0a28e85ac2fa","Type":"ContainerDied","Data":"8c3b49e3c97ceb4eee8b123262df0953029daae863b4fb248ab2f96290c0582c"} Oct 30 01:07:20.413258 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.413227 5300 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c3b49e3c97ceb4eee8b123262df0953029daae863b4fb248ab2f96290c0582c" Oct 30 01:07:20.433228 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.433196 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:20.446684 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.446655 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f" Oct 30 01:07:20.474954 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.474931 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257" Oct 30 01:07:20.527065 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.527023 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a22c9a38-2f90-46f2-b924-0a28e85ac2fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a22c9a38-2f90-46f2-b924-0a28e85ac2fa-plugins-config-volume\") pod \"a22c9a38-2f90-46f2-b924-0a28e85ac2fa\" (UID: \"a22c9a38-2f90-46f2-b924-0a28e85ac2fa\") " Oct 30 01:07:20.527269 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.527151 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a22c9a38-2f90-46f2-b924-0a28e85ac2fa-kube-api-access-tbkjd\" (UniqueName: \"kubernetes.io/projected/a22c9a38-2f90-46f2-b924-0a28e85ac2fa-kube-api-access-tbkjd\") pod \"a22c9a38-2f90-46f2-b924-0a28e85ac2fa\" (UID: \"a22c9a38-2f90-46f2-b924-0a28e85ac2fa\") " Oct 30 01:07:20.527269 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.527182 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37-kube-api-access-8zjbm\" (UniqueName: \"kubernetes.io/projected/fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37-kube-api-access-8zjbm\") pod \"fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37\" (UID: \"fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37\") " Oct 30 01:07:20.527422 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.527395 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37-plugins-config-volume\") pod \"fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37\" (UID: \"fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37\") " Oct 30 01:07:20.528126 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.528095 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a22c9a38-2f90-46f2-b924-0a28e85ac2fa-plugins-config-volume" pod "a22c9a38-2f90-46f2-b924-0a28e85ac2fa" (UID: "a22c9a38-2f90-46f2-b924-0a28e85ac2fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:07:20.528260 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.528156 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37-plugins-config-volume" pod "fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37" (UID: "fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:07:20.528260 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.528238 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a22c9a38-2f90-46f2-b924-0a28e85ac2fa-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:07:20.534837 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.531840 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22c9a38-2f90-46f2-b924-0a28e85ac2fa-kube-api-access-tbkjd" pod "a22c9a38-2f90-46f2-b924-0a28e85ac2fa" (UID: "a22c9a38-2f90-46f2-b924-0a28e85ac2fa"). InnerVolumeSpecName "kube-api-access-tbkjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:20.534837 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.532734 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37-kube-api-access-8zjbm" pod "fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37" (UID: "fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37"). InnerVolumeSpecName "kube-api-access-8zjbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:07:20.629831 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.629718 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbkjd\" (UniqueName: \"kubernetes.io/projected/a22c9a38-2f90-46f2-b924-0a28e85ac2fa-kube-api-access-tbkjd\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:07:20.629831 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.629746 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zjbm\" (UniqueName: \"kubernetes.io/projected/fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37-kube-api-access-8zjbm\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:07:20.629831 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.629756 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:07:20.705900 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.705864 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc"] Oct 30 01:07:20.706551 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.706537 5300 cpu_manager.go:414] "containerMap: removing container" podUID="fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37" containerName="payload-pre-processing" Oct 30 01:07:20.706615 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.706552 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37" containerName="payload-pre-processing" Oct 30 01:07:20.706615 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.706584 5300 cpu_manager.go:414] "containerMap: removing container" podUID="a22c9a38-2f90-46f2-b924-0a28e85ac2fa" containerName="payload-processing" Oct 30 01:07:20.706615 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.706589 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a22c9a38-2f90-46f2-b924-0a28e85ac2fa" containerName="payload-processing" Oct 30 01:07:20.706735 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.706726 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37" containerName="payload-pre-processing" Oct 30 01:07:20.706788 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.706741 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="a22c9a38-2f90-46f2-b924-0a28e85ac2fa" containerName="payload-processing" Oct 30 01:07:20.710486 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.710459 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc" Oct 30 01:07:20.728400 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.728368 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-07410c-serving-cert\"" Oct 30 01:07:20.729367 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.729342 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc"] Oct 30 01:07:20.784534 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.784495 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:20.833907 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.833870 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3394c8-e374-44aa-ae07-590de2d9c4bf-openshift-service-ca\") pod \"maas-api-e2e-api-b-07410c-685fb97884-wfzqc\" (UID: \"da3394c8-e374-44aa-ae07-590de2d9c4bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc" Oct 30 01:07:20.834099 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.834025 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da3394c8-e374-44aa-ae07-590de2d9c4bf-metrics-certs\") pod \"maas-api-e2e-api-b-07410c-685fb97884-wfzqc\" (UID: \"da3394c8-e374-44aa-ae07-590de2d9c4bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc" Oct 30 01:07:20.834230 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.834207 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da3394c8-e374-44aa-ae07-590de2d9c4bf-maas-api-tls\") pod \"maas-api-e2e-api-b-07410c-685fb97884-wfzqc\" (UID: \"da3394c8-e374-44aa-ae07-590de2d9c4bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc" Oct 30 01:07:20.834398 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.834336 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4gdt\" (UniqueName: \"kubernetes.io/projected/da3394c8-e374-44aa-ae07-590de2d9c4bf-kube-api-access-m4gdt\") pod \"maas-api-e2e-api-b-07410c-685fb97884-wfzqc\" (UID: \"da3394c8-e374-44aa-ae07-590de2d9c4bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc" Oct 30 01:07:20.866819 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.866791 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:20.935495 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.935416 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3394c8-e374-44aa-ae07-590de2d9c4bf-openshift-service-ca\") pod \"maas-api-e2e-api-b-07410c-685fb97884-wfzqc\" (UID: \"da3394c8-e374-44aa-ae07-590de2d9c4bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc" Oct 30 01:07:20.935495 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.935453 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da3394c8-e374-44aa-ae07-590de2d9c4bf-metrics-certs\") pod \"maas-api-e2e-api-b-07410c-685fb97884-wfzqc\" (UID: \"da3394c8-e374-44aa-ae07-590de2d9c4bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc" Oct 30 01:07:20.935727 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.935534 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da3394c8-e374-44aa-ae07-590de2d9c4bf-maas-api-tls\") pod \"maas-api-e2e-api-b-07410c-685fb97884-wfzqc\" (UID: \"da3394c8-e374-44aa-ae07-590de2d9c4bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc" Oct 30 01:07:20.935727 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.935579 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4gdt\" (UniqueName: \"kubernetes.io/projected/da3394c8-e374-44aa-ae07-590de2d9c4bf-kube-api-access-m4gdt\") pod \"maas-api-e2e-api-b-07410c-685fb97884-wfzqc\" (UID: \"da3394c8-e374-44aa-ae07-590de2d9c4bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc" Oct 30 01:07:20.936162 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.936139 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3394c8-e374-44aa-ae07-590de2d9c4bf-openshift-service-ca\") pod \"maas-api-e2e-api-b-07410c-685fb97884-wfzqc\" (UID: \"da3394c8-e374-44aa-ae07-590de2d9c4bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc" Oct 30 01:07:20.938128 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.938104 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da3394c8-e374-44aa-ae07-590de2d9c4bf-metrics-certs\") pod \"maas-api-e2e-api-b-07410c-685fb97884-wfzqc\" (UID: \"da3394c8-e374-44aa-ae07-590de2d9c4bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc" Oct 30 01:07:20.938222 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.938137 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da3394c8-e374-44aa-ae07-590de2d9c4bf-maas-api-tls\") pod \"maas-api-e2e-api-b-07410c-685fb97884-wfzqc\" (UID: \"da3394c8-e374-44aa-ae07-590de2d9c4bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc" Oct 30 01:07:20.956789 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.956724 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4gdt\" (UniqueName: \"kubernetes.io/projected/da3394c8-e374-44aa-ae07-590de2d9c4bf-kube-api-access-m4gdt\") pod \"maas-api-e2e-api-b-07410c-685fb97884-wfzqc\" (UID: \"da3394c8-e374-44aa-ae07-590de2d9c4bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc" Oct 30 01:07:20.967506 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:20.967477 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:21.027636 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:21.027597 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc" Oct 30 01:07:21.072379 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:21.072346 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:21.362268 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:07:21.362240 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3394c8_e374_44aa_ae07_590de2d9c4bf.slice/crio-c099689f4c5f5c7426cf035da6c740eec17271745adf53c460d697ea8e352c2a WatchSource:0}: Error finding container c099689f4c5f5c7426cf035da6c740eec17271745adf53c460d697ea8e352c2a: Status 404 returned error can't find the container with id c099689f4c5f5c7426cf035da6c740eec17271745adf53c460d697ea8e352c2a Oct 30 01:07:21.362891 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:21.362857 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc"] Oct 30 01:07:21.425878 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:21.425838 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257" event={"ID":"fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37","Type":"ContainerDied","Data":"613103df30b6730c92af27e0a506e3c80f2deb3882147238a7147fbe73f5c806"} Oct 30 01:07:21.426048 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:21.425918 5300 scope.go:122] "RemoveContainer" containerID="3301c565b1809fb85cad92634ee899d886520b2c2b8ceeb3bd2316e7f945111a" Oct 30 01:07:21.426048 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:21.425848 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257" Oct 30 01:07:21.427720 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:21.427654 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc" event={"ID":"da3394c8-e374-44aa-ae07-590de2d9c4bf","Type":"ContainerStarted","Data":"c099689f4c5f5c7426cf035da6c740eec17271745adf53c460d697ea8e352c2a"} Oct 30 01:07:21.427826 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:21.427778 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f" Oct 30 01:07:21.447042 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:21.447009 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:21.503508 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:21.503472 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257"] Oct 30 01:07:21.517258 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:21.517221 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-17d855-7579cb887-tl257"] Oct 30 01:07:21.545977 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:21.545936 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f"] Oct 30 01:07:21.549457 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:21.549431 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-17d855-7cd78798fd-tdv5f"] Oct 30 01:07:21.598745 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:07:21.598709 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:07:21.826411 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:21.826386 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a22c9a38-2f90-46f2-b924-0a28e85ac2fa" path="/var/lib/kubelet/pods/a22c9a38-2f90-46f2-b924-0a28e85ac2fa/volumes" Oct 30 01:07:21.826908 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:21.826884 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37" path="/var/lib/kubelet/pods/fa4a3b1f-1bd3-4a67-b2c3-f8db8f241d37/volumes" Oct 30 01:07:21.951231 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:21.951197 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:22.175550 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:22.175427 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:22.441799 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:22.441636 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc" event={"ID":"da3394c8-e374-44aa-ae07-590de2d9c4bf","Type":"ContainerStarted","Data":"1ee2c5f9ed483a68d76bfbf43de03d26cd1268aaa7e7b75ef336d643587979cc"} Oct 30 01:07:22.442948 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:22.442910 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc" Oct 30 01:07:22.487605 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:22.487547 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc" podStartSLOduration=2.029475239 podStartE2EDuration="2.487533814s" totalImagesPullingTime="458.058575ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:22.484980545 +0000 UTC m=+1442.800848870" watchObservedRunningTime="2026-10-30 01:07:22.487533814 +0000 UTC m=+1442.803402139" Oct 30 01:07:22.732531 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:22.732497 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs"] Oct 30 01:07:22.737125 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:22.737100 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs" Oct 30 01:07:22.755405 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:22.755377 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs"] Oct 30 01:07:22.789592 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:22.789556 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-13a3e2-serving-cert\"" Oct 30 01:07:22.820474 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:22.820407 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:22.886924 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:22.886886 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42434c3a-b364-4693-8493-c77ae7baf016-metrics-certs\") pod \"maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs\" (UID: \"42434c3a-b364-4693-8493-c77ae7baf016\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs" Oct 30 01:07:22.887097 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:22.887024 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42434c3a-b364-4693-8493-c77ae7baf016-maas-api-tls\") pod \"maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs\" (UID: \"42434c3a-b364-4693-8493-c77ae7baf016\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs" Oct 30 01:07:22.887239 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:22.887219 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42434c3a-b364-4693-8493-c77ae7baf016-openshift-service-ca\") pod \"maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs\" (UID: \"42434c3a-b364-4693-8493-c77ae7baf016\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs" Oct 30 01:07:22.887326 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:22.887312 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72pgb\" (UniqueName: \"kubernetes.io/projected/42434c3a-b364-4693-8493-c77ae7baf016-kube-api-access-72pgb\") pod \"maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs\" (UID: \"42434c3a-b364-4693-8493-c77ae7baf016\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs" Oct 30 01:07:22.919693 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:22.919661 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:22.988566 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:22.988477 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42434c3a-b364-4693-8493-c77ae7baf016-openshift-service-ca\") pod \"maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs\" (UID: \"42434c3a-b364-4693-8493-c77ae7baf016\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs" Oct 30 01:07:22.988566 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:22.988542 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72pgb\" (UniqueName: \"kubernetes.io/projected/42434c3a-b364-4693-8493-c77ae7baf016-kube-api-access-72pgb\") pod \"maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs\" (UID: \"42434c3a-b364-4693-8493-c77ae7baf016\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs" Oct 30 01:07:22.988810 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:22.988638 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42434c3a-b364-4693-8493-c77ae7baf016-metrics-certs\") pod \"maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs\" (UID: \"42434c3a-b364-4693-8493-c77ae7baf016\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs" Oct 30 01:07:22.988810 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:22.988703 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42434c3a-b364-4693-8493-c77ae7baf016-maas-api-tls\") pod \"maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs\" (UID: \"42434c3a-b364-4693-8493-c77ae7baf016\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs" Oct 30 01:07:22.990200 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:22.989306 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42434c3a-b364-4693-8493-c77ae7baf016-openshift-service-ca\") pod \"maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs\" (UID: \"42434c3a-b364-4693-8493-c77ae7baf016\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs" Oct 30 01:07:22.991371 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:22.991347 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42434c3a-b364-4693-8493-c77ae7baf016-metrics-certs\") pod \"maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs\" (UID: \"42434c3a-b364-4693-8493-c77ae7baf016\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs" Oct 30 01:07:22.991502 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:22.991420 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42434c3a-b364-4693-8493-c77ae7baf016-maas-api-tls\") pod \"maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs\" (UID: \"42434c3a-b364-4693-8493-c77ae7baf016\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs" Oct 30 01:07:23.010124 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:23.010086 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72pgb\" (UniqueName: \"kubernetes.io/projected/42434c3a-b364-4693-8493-c77ae7baf016-kube-api-access-72pgb\") pod \"maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs\" (UID: \"42434c3a-b364-4693-8493-c77ae7baf016\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs" Oct 30 01:07:23.018252 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:23.018221 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:23.052373 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:23.052340 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs" Oct 30 01:07:23.367250 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:07:23.367222 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42434c3a_b364_4693_8493_c77ae7baf016.slice/crio-3838321a78f329d6a376ad04c38a66dd95fbc1ab63bfe9ed083486921ce2d67f WatchSource:0}: Error finding container 3838321a78f329d6a376ad04c38a66dd95fbc1ab63bfe9ed083486921ce2d67f: Status 404 returned error can't find the container with id 3838321a78f329d6a376ad04c38a66dd95fbc1ab63bfe9ed083486921ce2d67f Oct 30 01:07:23.367664 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:23.367640 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs"] Oct 30 01:07:23.449944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:23.449902 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs" event={"ID":"42434c3a-b364-4693-8493-c77ae7baf016","Type":"ContainerStarted","Data":"3838321a78f329d6a376ad04c38a66dd95fbc1ab63bfe9ed083486921ce2d67f"} Oct 30 01:07:24.459161 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:24.459125 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs" event={"ID":"42434c3a-b364-4693-8493-c77ae7baf016","Type":"ContainerStarted","Data":"3ee0ef7a3b7da0e8db55c012a938215ab92af70ec981f672371fef06ad97ec30"} Oct 30 01:07:24.459645 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:24.459221 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs" Oct 30 01:07:24.516317 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:24.516255 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs" podStartSLOduration=2.21756132 podStartE2EDuration="2.516234555s" totalImagesPullingTime="298.673235ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:24.512455546 +0000 UTC m=+1444.828323872" watchObservedRunningTime="2026-10-30 01:07:24.516234555 +0000 UTC m=+1444.832102881" Oct 30 01:07:24.587398 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:24.587356 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:26.281666 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:26.281633 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:26.723079 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:26.723043 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:26.836568 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:26.836533 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:28.198814 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:28.198783 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:28.437592 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:28.437555 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:28.456333 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:28.456258 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc" Oct 30 01:07:28.541555 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:28.541519 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:30.347567 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:30.347478 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:30.475516 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:30.475487 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs" Oct 30 01:07:32.260409 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:32.260300 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:32.369028 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:32.368989 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:33.852023 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:33.851989 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:35.617173 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:35.617131 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:37.238141 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:37.238107 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:38.788403 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:38.788343 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:38.918693 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:38.918653 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:39.042510 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:39.042409 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:40.014171 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:40.014141 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:40.116382 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:40.116349 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:40.214060 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:40.214027 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:40.668172 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:40.668136 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:40.755993 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:40.755961 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:42.505479 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:42.505443 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:44.268023 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:44.267991 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:45.908513 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:45.908472 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:46.017878 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:46.010155 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:46.107060 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:46.107025 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:47.754662 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:47.754626 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:49.415619 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:49.415583 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:51.168096 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:51.168054 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:51.594461 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:07:51.594419 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:07:52.133709 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:52.133674 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:53.277507 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:53.277474 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:55.151389 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:55.151343 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:56.774145 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:56.774087 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:58.175877 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:58.175827 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:07:58.574530 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:07:58.574196 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:00.619932 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:00.619895 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:00.700230 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:00.700193 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:00.911245 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:00.911156 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:01.012235 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:01.012191 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:02.131181 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:02.131147 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r"] Oct 30 01:08:02.131691 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:02.131460 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r" podUID="a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4" containerName="maas-api" containerID="cri-o://0c9d268d6aaa208c8e34b4d6ace6e360c46c36e44bd363acf70a7c85d30e7d1b" gracePeriod=30 Oct 30 01:08:02.213216 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:02.213169 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:02.616167 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:02.616142 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r" Oct 30 01:08:02.650527 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:02.650481 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-kube-api-access-78lbh\" (UniqueName: \"kubernetes.io/projected/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-kube-api-access-78lbh\") pod \"a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4\" (UID: \"a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4\") " Oct 30 01:08:02.650725 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:02.650647 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-openshift-service-ca\") pod \"a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4\" (UID: \"a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4\") " Oct 30 01:08:02.650821 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:02.650808 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-maas-api-tls\") pod \"a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4\" (UID: \"a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4\") " Oct 30 01:08:02.650973 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:02.650955 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-metrics-certs\") pod \"a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4\" (UID: \"a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4\") " Oct 30 01:08:02.673326 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:02.657040 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-openshift-service-ca" pod "a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4" (UID: "a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:08:02.682832 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:02.678775 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-metrics-certs" pod "a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4" (UID: "a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:08:02.683118 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:02.682959 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-kube-api-access-78lbh" pod "a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4" (UID: "a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4"). InnerVolumeSpecName "kube-api-access-78lbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:02.694886 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:02.686744 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-maas-api-tls" pod "a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4" (UID: "a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:08:02.759838 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:02.753032 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-maas-api-tls\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:08:02.759838 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:02.753071 5300 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-metrics-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:08:02.759838 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:02.753086 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78lbh\" (UniqueName: \"kubernetes.io/projected/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-kube-api-access-78lbh\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:08:02.759838 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:02.753103 5300 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4-openshift-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:08:02.838711 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:02.838673 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:02.852775 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:02.852706 5300 generic.go:356] "Generic (PLEG): container finished" podID="a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4" containerID="0c9d268d6aaa208c8e34b4d6ace6e360c46c36e44bd363acf70a7c85d30e7d1b" exitCode=0 Oct 30 01:08:02.852984 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:02.852937 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r" event={"ID":"a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4","Type":"ContainerDied","Data":"0c9d268d6aaa208c8e34b4d6ace6e360c46c36e44bd363acf70a7c85d30e7d1b"} Oct 30 01:08:02.853066 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:02.852990 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r" event={"ID":"a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4","Type":"ContainerDied","Data":"182de75c27e0571afe461b6fa1d3a494c59ae56f1a5251577a20b5bad3b01c71"} Oct 30 01:08:02.853066 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:02.853020 5300 scope.go:122] "RemoveContainer" containerID="0c9d268d6aaa208c8e34b4d6ace6e360c46c36e44bd363acf70a7c85d30e7d1b" Oct 30 01:08:02.853191 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:02.853184 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r" Oct 30 01:08:02.875406 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:02.875373 5300 scope.go:122] "RemoveContainer" containerID="0c9d268d6aaa208c8e34b4d6ace6e360c46c36e44bd363acf70a7c85d30e7d1b" Oct 30 01:08:02.875975 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:08:02.875945 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c9d268d6aaa208c8e34b4d6ace6e360c46c36e44bd363acf70a7c85d30e7d1b\": container with ID starting with 0c9d268d6aaa208c8e34b4d6ace6e360c46c36e44bd363acf70a7c85d30e7d1b not found: ID does not exist" containerID="0c9d268d6aaa208c8e34b4d6ace6e360c46c36e44bd363acf70a7c85d30e7d1b" Oct 30 01:08:02.876087 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:02.875982 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c9d268d6aaa208c8e34b4d6ace6e360c46c36e44bd363acf70a7c85d30e7d1b"} err="failed to get container status \"0c9d268d6aaa208c8e34b4d6ace6e360c46c36e44bd363acf70a7c85d30e7d1b\": rpc error: code = NotFound desc = could not find container \"0c9d268d6aaa208c8e34b4d6ace6e360c46c36e44bd363acf70a7c85d30e7d1b\": container with ID starting with 0c9d268d6aaa208c8e34b4d6ace6e360c46c36e44bd363acf70a7c85d30e7d1b not found: ID does not exist" Oct 30 01:08:02.915697 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:02.915664 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r"] Oct 30 01:08:02.919565 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:02.919528 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-17d855-687c555c94-56z5r"] Oct 30 01:08:02.928286 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:02.928252 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:03.227622 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:03.227591 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:03.328183 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:03.328102 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:03.564640 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:03.564603 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:03.662146 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:03.662106 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:03.824941 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:03.824856 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4" path="/var/lib/kubelet/pods/a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4/volumes" Oct 30 01:08:03.863057 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:03.863025 5300 generic.go:356] "Generic (PLEG): container finished" podID="c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2" containerID="f916fe3c48aaad148fdacf849fa41dc4e50ea86dc4e2d80e52712f4f047d6ce4" exitCode=0 Oct 30 01:08:03.863296 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:03.863099 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" event={"ID":"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2","Type":"ContainerDied","Data":"f916fe3c48aaad148fdacf849fa41dc4e50ea86dc4e2d80e52712f4f047d6ce4"} Oct 30 01:08:03.863629 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:03.863610 5300 scope.go:122] "RemoveContainer" containerID="f916fe3c48aaad148fdacf849fa41dc4e50ea86dc4e2d80e52712f4f047d6ce4" Oct 30 01:08:03.865496 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:03.865461 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:08:03.865597 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:03.865553 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:08:03.865659 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:03.865606 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:08:04.034703 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:04.034668 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:04.635339 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:04.635300 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:04.871568 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:04.871538 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" event={"ID":"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2","Type":"ContainerStarted","Data":"7aec454b3ac8709091367196e5282c60b7d4d18d97b3b9e4b44306297e302743"} Oct 30 01:08:04.958625 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:04.958533 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:04.971416 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:04.971384 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:08:04.972654 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:04.972627 5300 patch_prober.go:29] interesting pod/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 01:08:04.972895 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:04.972685 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" podUID="c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 01:08:05.084004 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:05.083960 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:08:05.971654 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:05.971617 5300 patch_prober.go:29] interesting pod/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 01:08:05.972114 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:05.971672 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" podUID="c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 01:08:06.971869 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:06.971827 5300 patch_prober.go:29] interesting pod/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 01:08:06.972880 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:06.972595 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" podUID="c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 01:08:06.985749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:06.985717 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:07.936231 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:07.936192 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:07.971235 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:07.971201 5300 patch_prober.go:29] interesting pod/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 01:08:07.971408 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:07.971284 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" podUID="c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 01:08:08.048292 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:08.048260 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:08.971441 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:08.971403 5300 patch_prober.go:29] interesting pod/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 01:08:08.971619 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:08.971474 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" podUID="c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 01:08:09.169503 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:09.169469 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:09.675476 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:09.675435 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:09.778026 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:09.777988 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:09.871450 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:09.871418 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc"] Oct 30 01:08:09.872754 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:09.872725 5300 cpu_manager.go:414] "containerMap: removing container" podUID="a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4" containerName="maas-api" Oct 30 01:08:09.872754 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:09.872753 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4" containerName="maas-api" Oct 30 01:08:09.872979 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:09.872965 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0f0c6c2-021f-4b5b-9f4f-5d1f64bd76d4" containerName="maas-api" Oct 30 01:08:09.877139 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:09.877119 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc"] Oct 30 01:08:09.877248 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:09.877201 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc" Oct 30 01:08:09.893990 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:09.893958 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-da60de\"" Oct 30 01:08:09.908169 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:09.908141 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-da60de-dockercfg-5f5z5\"" Oct 30 01:08:09.974063 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:09.974016 5300 patch_prober.go:29] interesting pod/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 01:08:09.974271 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:09.974076 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" podUID="c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 01:08:09.996782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:09.992294 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:10.020789 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:10.019895 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dee7a3e4-4f8c-41d0-b445-6c40118a8ae0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc\" (UID: \"dee7a3e4-4f8c-41d0-b445-6c40118a8ae0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc" Oct 30 01:08:10.020789 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:10.020076 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j7mn\" (UniqueName: \"kubernetes.io/projected/dee7a3e4-4f8c-41d0-b445-6c40118a8ae0-kube-api-access-4j7mn\") pod \"payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc\" (UID: \"dee7a3e4-4f8c-41d0-b445-6c40118a8ae0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc" Oct 30 01:08:10.100503 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:10.100466 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:10.135011 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:10.134973 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dee7a3e4-4f8c-41d0-b445-6c40118a8ae0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc\" (UID: \"dee7a3e4-4f8c-41d0-b445-6c40118a8ae0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc" Oct 30 01:08:10.149436 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:10.149391 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j7mn\" (UniqueName: \"kubernetes.io/projected/dee7a3e4-4f8c-41d0-b445-6c40118a8ae0-kube-api-access-4j7mn\") pod \"payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc\" (UID: \"dee7a3e4-4f8c-41d0-b445-6c40118a8ae0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc" Oct 30 01:08:10.150526 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:10.136089 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dee7a3e4-4f8c-41d0-b445-6c40118a8ae0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc\" (UID: \"dee7a3e4-4f8c-41d0-b445-6c40118a8ae0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc" Oct 30 01:08:10.189824 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:10.188702 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j7mn\" (UniqueName: \"kubernetes.io/projected/dee7a3e4-4f8c-41d0-b445-6c40118a8ae0-kube-api-access-4j7mn\") pod \"payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc\" (UID: \"dee7a3e4-4f8c-41d0-b445-6c40118a8ae0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc" Oct 30 01:08:10.193111 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:10.193077 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc" Oct 30 01:08:10.368704 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:10.356541 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:10.634287 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:10.631085 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc"] Oct 30 01:08:10.940798 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:10.940655 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc" event={"ID":"dee7a3e4-4f8c-41d0-b445-6c40118a8ae0","Type":"ContainerStarted","Data":"8909528784b1df69d9f0c3e5f4d026c8ed5c59d099333ffd4e1a1b859696705b"} Oct 30 01:08:10.940798 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:10.940723 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc" event={"ID":"dee7a3e4-4f8c-41d0-b445-6c40118a8ae0","Type":"ContainerStarted","Data":"3e733a5e5c8f7a71d9aa39922ec35b80f85e97a804fc16077d1fa8a7ccbc4b95"} Oct 30 01:08:10.941080 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:10.941056 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc" Oct 30 01:08:10.979328 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:10.978248 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:08:10.982069 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:10.981975 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:08:11.020457 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:11.020386 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc" podStartSLOduration=2.02036109 podStartE2EDuration="2.02036109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:10.99004859 +0000 UTC m=+1491.305916914" watchObservedRunningTime="2026-10-30 01:08:11.02036109 +0000 UTC m=+1491.336229417" Oct 30 01:08:11.340323 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:11.340292 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:13.096030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:13.095996 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:13.301873 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:13.301816 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:13.413856 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:13.411246 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:15.169878 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:15.169835 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:17.251786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:17.247395 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:19.293786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:19.285780 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:19.311786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:19.304872 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:08:19.311786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:19.306265 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:08:19.311786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:19.306572 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" podUID="d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 01:08:19.311786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:19.306849 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:08:19.663847 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:19.663571 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:08:19.695136 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:19.692850 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 30 01:08:21.377348 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:21.377315 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:21.629002 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:08:21.628906 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:08:21.952755 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:21.952656 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc" Oct 30 01:08:22.516546 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:22.516510 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn"] Oct 30 01:08:22.579817 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:22.579782 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5"] Oct 30 01:08:22.580066 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:22.580041 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" podUID="443d5afc-4682-442b-9dbb-9e125c2973a0" containerName="istio-proxy" containerID="cri-o://ed8defd10942afc601c1cd69f3b790fa972e49ad971bf514202ad1920f8075dd" gracePeriod=30 Oct 30 01:08:22.656620 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:22.656581 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:22.752480 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:22.752432 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:23.398340 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:23.398278 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:23.830487 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:23.830450 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:25.232393 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:25.232352 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:26.160729 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:26.160698 5300 generic.go:356] "Generic (PLEG): container finished" podID="d13883c2-ec67-4526-985d-f22c6673853f" containerID="1a8a522e7b395c6e2a170ad1cfee67b899241e218c1d080eae472f64f7b5527f" exitCode=0 Oct 30 01:08:26.160979 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:26.160781 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" event={"ID":"d13883c2-ec67-4526-985d-f22c6673853f","Type":"ContainerDied","Data":"1a8a522e7b395c6e2a170ad1cfee67b899241e218c1d080eae472f64f7b5527f"} Oct 30 01:08:26.161292 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:26.161265 5300 scope.go:122] "RemoveContainer" containerID="1a8a522e7b395c6e2a170ad1cfee67b899241e218c1d080eae472f64f7b5527f" Oct 30 01:08:26.163108 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:26.163080 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:08:26.163197 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:26.163153 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:08:26.163197 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:26.163188 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:08:27.162816 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:27.162740 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:27.170451 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:27.170417 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" event={"ID":"d13883c2-ec67-4526-985d-f22c6673853f","Type":"ContainerStarted","Data":"970d0565450e21b0ef7693b3afc397a52d15f09b9a33c0954732b2b1f66d1d6a"} Oct 30 01:08:27.322251 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:27.322218 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:08:27.517162 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:27.517122 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn" podUID="8c9209c7-2686-4759-96f2-efb5092b4536" containerName="payload-processing" containerID="cri-o://57ab1ec66ceee954d9db0a1c4c1d780ab5d6bf5bd254760438dae80c79225d7a" gracePeriod=25 Oct 30 01:08:27.685011 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:27.684966 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:08:27.686079 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:27.686052 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:08:27.686207 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:27.686124 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" podUID="d13883c2-ec67-4526-985d-f22c6673853f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 01:08:27.917470 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:27.917434 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn" Oct 30 01:08:27.997752 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:27.997719 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c9209c7-2686-4759-96f2-efb5092b4536-kube-api-access-gdwxl\" (UniqueName: \"kubernetes.io/projected/8c9209c7-2686-4759-96f2-efb5092b4536-kube-api-access-gdwxl\") pod \"8c9209c7-2686-4759-96f2-efb5092b4536\" (UID: \"8c9209c7-2686-4759-96f2-efb5092b4536\") " Oct 30 01:08:27.997987 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:27.997961 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c9209c7-2686-4759-96f2-efb5092b4536-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c9209c7-2686-4759-96f2-efb5092b4536-plugins-config-volume\") pod \"8c9209c7-2686-4759-96f2-efb5092b4536\" (UID: \"8c9209c7-2686-4759-96f2-efb5092b4536\") " Oct 30 01:08:27.999464 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:27.999435 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c9209c7-2686-4759-96f2-efb5092b4536-plugins-config-volume" pod "8c9209c7-2686-4759-96f2-efb5092b4536" (UID: "8c9209c7-2686-4759-96f2-efb5092b4536"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:08:28.001570 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.001542 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c9209c7-2686-4759-96f2-efb5092b4536-kube-api-access-gdwxl" pod "8c9209c7-2686-4759-96f2-efb5092b4536" (UID: "8c9209c7-2686-4759-96f2-efb5092b4536"). InnerVolumeSpecName "kube-api-access-gdwxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:28.051297 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.051222 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 01:08:28.101040 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.101004 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c9209c7-2686-4759-96f2-efb5092b4536-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:08:28.101040 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.101042 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdwxl\" (UniqueName: \"kubernetes.io/projected/8c9209c7-2686-4759-96f2-efb5092b4536-kube-api-access-gdwxl\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:08:28.180921 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.180882 5300 generic.go:356] "Generic (PLEG): container finished" podID="8c9209c7-2686-4759-96f2-efb5092b4536" containerID="57ab1ec66ceee954d9db0a1c4c1d780ab5d6bf5bd254760438dae80c79225d7a" exitCode=0 Oct 30 01:08:28.181358 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.180935 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn" Oct 30 01:08:28.181358 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.180961 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn" event={"ID":"8c9209c7-2686-4759-96f2-efb5092b4536","Type":"ContainerDied","Data":"57ab1ec66ceee954d9db0a1c4c1d780ab5d6bf5bd254760438dae80c79225d7a"} Oct 30 01:08:28.181358 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.181013 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn" event={"ID":"8c9209c7-2686-4759-96f2-efb5092b4536","Type":"ContainerDied","Data":"6641fa912137cc2aada56552dc6d187a10c363db829471b73aa13071b5acb4e0"} Oct 30 01:08:28.181358 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.181039 5300 scope.go:122] "RemoveContainer" containerID="57ab1ec66ceee954d9db0a1c4c1d780ab5d6bf5bd254760438dae80c79225d7a" Oct 30 01:08:28.182769 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.182733 5300 generic.go:356] "Generic (PLEG): container finished" podID="443d5afc-4682-442b-9dbb-9e125c2973a0" containerID="ed8defd10942afc601c1cd69f3b790fa972e49ad971bf514202ad1920f8075dd" exitCode=0 Oct 30 01:08:28.182897 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.182794 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" Oct 30 01:08:28.182897 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.182857 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" event={"ID":"443d5afc-4682-442b-9dbb-9e125c2973a0","Type":"ContainerDied","Data":"ed8defd10942afc601c1cd69f3b790fa972e49ad971bf514202ad1920f8075dd"} Oct 30 01:08:28.183026 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.182911 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5" event={"ID":"443d5afc-4682-442b-9dbb-9e125c2973a0","Type":"ContainerDied","Data":"1a2c2168656980714b2cceff19d44c6c1af0cc984e5a0b4de9a267dabca45121"} Oct 30 01:08:28.192996 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.192973 5300 scope.go:122] "RemoveContainer" containerID="57ab1ec66ceee954d9db0a1c4c1d780ab5d6bf5bd254760438dae80c79225d7a" Oct 30 01:08:28.193346 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:08:28.193323 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57ab1ec66ceee954d9db0a1c4c1d780ab5d6bf5bd254760438dae80c79225d7a\": container with ID starting with 57ab1ec66ceee954d9db0a1c4c1d780ab5d6bf5bd254760438dae80c79225d7a not found: ID does not exist" containerID="57ab1ec66ceee954d9db0a1c4c1d780ab5d6bf5bd254760438dae80c79225d7a" Oct 30 01:08:28.193421 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.193354 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57ab1ec66ceee954d9db0a1c4c1d780ab5d6bf5bd254760438dae80c79225d7a"} err="failed to get container status \"57ab1ec66ceee954d9db0a1c4c1d780ab5d6bf5bd254760438dae80c79225d7a\": rpc error: code = NotFound desc = could not find container \"57ab1ec66ceee954d9db0a1c4c1d780ab5d6bf5bd254760438dae80c79225d7a\": container with ID starting with 57ab1ec66ceee954d9db0a1c4c1d780ab5d6bf5bd254760438dae80c79225d7a not found: ID does not exist" Oct 30 01:08:28.193421 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.193370 5300 scope.go:122] "RemoveContainer" containerID="ed8defd10942afc601c1cd69f3b790fa972e49ad971bf514202ad1920f8075dd" Oct 30 01:08:28.202070 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.202042 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-envoy\") pod \"443d5afc-4682-442b-9dbb-9e125c2973a0\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " Oct 30 01:08:28.202168 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.202094 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-workload-certs\") pod \"443d5afc-4682-442b-9dbb-9e125c2973a0\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " Oct 30 01:08:28.202168 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.202127 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/443d5afc-4682-442b-9dbb-9e125c2973a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/443d5afc-4682-442b-9dbb-9e125c2973a0-istiod-ca-cert\") pod \"443d5afc-4682-442b-9dbb-9e125c2973a0\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " Oct 30 01:08:28.202263 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.202228 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-podinfo\") pod \"443d5afc-4682-442b-9dbb-9e125c2973a0\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " Oct 30 01:08:28.202319 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.202280 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/443d5afc-4682-442b-9dbb-9e125c2973a0-kube-api-access-rrbd2\" (UniqueName: \"kubernetes.io/projected/443d5afc-4682-442b-9dbb-9e125c2973a0-kube-api-access-rrbd2\") pod \"443d5afc-4682-442b-9dbb-9e125c2973a0\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " Oct 30 01:08:28.203794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.202414 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-workload-certs" pod "443d5afc-4682-442b-9dbb-9e125c2973a0" (UID: "443d5afc-4682-442b-9dbb-9e125c2973a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:28.203794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.202424 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-credential-socket\") pod \"443d5afc-4682-442b-9dbb-9e125c2973a0\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " Oct 30 01:08:28.203794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.202513 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-data\") pod \"443d5afc-4682-442b-9dbb-9e125c2973a0\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " Oct 30 01:08:28.203794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.202630 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-credential-socket" pod "443d5afc-4682-442b-9dbb-9e125c2973a0" (UID: "443d5afc-4682-442b-9dbb-9e125c2973a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:28.203794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.202638 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-token\" (UniqueName: \"kubernetes.io/projected/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-token\") pod \"443d5afc-4682-442b-9dbb-9e125c2973a0\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " Oct 30 01:08:28.203794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.202731 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-workload-socket\") pod \"443d5afc-4682-442b-9dbb-9e125c2973a0\" (UID: \"443d5afc-4682-442b-9dbb-9e125c2973a0\") " Oct 30 01:08:28.203794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.202770 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/443d5afc-4682-442b-9dbb-9e125c2973a0-istiod-ca-cert" pod "443d5afc-4682-442b-9dbb-9e125c2973a0" (UID: "443d5afc-4682-442b-9dbb-9e125c2973a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:08:28.203794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.203361 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-data" pod "443d5afc-4682-442b-9dbb-9e125c2973a0" (UID: "443d5afc-4682-442b-9dbb-9e125c2973a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:28.203794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.203550 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-workload-socket" pod "443d5afc-4682-442b-9dbb-9e125c2973a0" (UID: "443d5afc-4682-442b-9dbb-9e125c2973a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:28.203794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.203565 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:08:28.203794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.203589 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/443d5afc-4682-442b-9dbb-9e125c2973a0-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:08:28.203794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.203603 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:08:28.203794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.203616 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:08:28.210740 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.208838 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-envoy" pod "443d5afc-4682-442b-9dbb-9e125c2973a0" (UID: "443d5afc-4682-442b-9dbb-9e125c2973a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:08:28.213110 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.213026 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-podinfo" pod "443d5afc-4682-442b-9dbb-9e125c2973a0" (UID: "443d5afc-4682-442b-9dbb-9e125c2973a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:08:28.213895 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.213747 5300 scope.go:122] "RemoveContainer" containerID="ed8defd10942afc601c1cd69f3b790fa972e49ad971bf514202ad1920f8075dd" Oct 30 01:08:28.214184 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:08:28.214139 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed8defd10942afc601c1cd69f3b790fa972e49ad971bf514202ad1920f8075dd\": container with ID starting with ed8defd10942afc601c1cd69f3b790fa972e49ad971bf514202ad1920f8075dd not found: ID does not exist" containerID="ed8defd10942afc601c1cd69f3b790fa972e49ad971bf514202ad1920f8075dd" Oct 30 01:08:28.214311 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.214178 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed8defd10942afc601c1cd69f3b790fa972e49ad971bf514202ad1920f8075dd"} err="failed to get container status \"ed8defd10942afc601c1cd69f3b790fa972e49ad971bf514202ad1920f8075dd\": rpc error: code = NotFound desc = could not find container \"ed8defd10942afc601c1cd69f3b790fa972e49ad971bf514202ad1920f8075dd\": container with ID starting with ed8defd10942afc601c1cd69f3b790fa972e49ad971bf514202ad1920f8075dd not found: ID does not exist" Oct 30 01:08:28.214438 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.214416 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/443d5afc-4682-442b-9dbb-9e125c2973a0-kube-api-access-rrbd2" pod "443d5afc-4682-442b-9dbb-9e125c2973a0" (UID: "443d5afc-4682-442b-9dbb-9e125c2973a0"). InnerVolumeSpecName "kube-api-access-rrbd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:28.214491 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.214475 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-token" pod "443d5afc-4682-442b-9dbb-9e125c2973a0" (UID: "443d5afc-4682-442b-9dbb-9e125c2973a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:28.257857 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.257822 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn"] Oct 30 01:08:28.261392 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.261367 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-288da6-b54cbc654-wx2zn"] Oct 30 01:08:28.309372 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.305738 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:08:28.309372 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.305783 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:08:28.309372 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.305799 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:08:28.309372 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.305818 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/443d5afc-4682-442b-9dbb-9e125c2973a0-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:08:28.309372 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.305832 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrbd2\" (UniqueName: \"kubernetes.io/projected/443d5afc-4682-442b-9dbb-9e125c2973a0-kube-api-access-rrbd2\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:08:28.546233 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.546190 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5"] Oct 30 01:08:28.554426 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.554386 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-28fc5"] Oct 30 01:08:28.684823 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.684751 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:08:28.685014 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:28.684842 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" podUID="d13883c2-ec67-4526-985d-f22c6673853f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 01:08:29.317013 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:29.316979 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:29.442781 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:29.434046 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:29.685040 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:29.684959 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:08:29.685040 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:29.685010 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" podUID="d13883c2-ec67-4526-985d-f22c6673853f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 01:08:29.754843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:29.754811 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:29.867470 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:29.867431 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="443d5afc-4682-442b-9dbb-9e125c2973a0" path="/var/lib/kubelet/pods/443d5afc-4682-442b-9dbb-9e125c2973a0/volumes" Oct 30 01:08:29.868201 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:29.868172 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c9209c7-2686-4759-96f2-efb5092b4536" path="/var/lib/kubelet/pods/8c9209c7-2686-4759-96f2-efb5092b4536/volumes" Oct 30 01:08:30.244784 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:30.244401 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:30.681237 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:30.681130 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:30.684752 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:30.684722 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:08:30.684752 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:30.684819 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" podUID="d13883c2-ec67-4526-985d-f22c6673853f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 01:08:31.169476 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:31.169442 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:31.685159 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:31.685126 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:08:31.685584 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:31.685190 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" podUID="d13883c2-ec67-4526-985d-f22c6673853f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 01:08:32.397579 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:32.397536 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:32.685475 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:32.685387 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:08:32.685475 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:32.685448 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" podUID="d13883c2-ec67-4526-985d-f22c6673853f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 01:08:33.071856 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:33.071819 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:33.690582 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:33.690550 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:08:33.691850 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:33.691826 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:08:34.051324 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:34.051285 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:34.167538 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:34.167499 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:34.983324 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:34.983283 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:35.781210 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:35.781177 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:35.880949 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:35.880907 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:35.979715 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:35.979676 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw"] Oct 30 01:08:35.980484 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:35.980462 5300 cpu_manager.go:414] "containerMap: removing container" podUID="443d5afc-4682-442b-9dbb-9e125c2973a0" containerName="istio-proxy" Oct 30 01:08:35.980484 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:35.980485 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="443d5afc-4682-442b-9dbb-9e125c2973a0" containerName="istio-proxy" Oct 30 01:08:35.980662 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:35.980523 5300 cpu_manager.go:414] "containerMap: removing container" podUID="8c9209c7-2686-4759-96f2-efb5092b4536" containerName="payload-processing" Oct 30 01:08:35.980662 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:35.980531 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c9209c7-2686-4759-96f2-efb5092b4536" containerName="payload-processing" Oct 30 01:08:35.980778 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:35.980735 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="443d5afc-4682-442b-9dbb-9e125c2973a0" containerName="istio-proxy" Oct 30 01:08:35.980778 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:35.980773 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c9209c7-2686-4759-96f2-efb5092b4536" containerName="payload-processing" Oct 30 01:08:35.986112 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:35.986087 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw" Oct 30 01:08:36.001519 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:36.001486 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw"] Oct 30 01:08:36.011346 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:36.011311 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-add1bb\"" Oct 30 01:08:36.011539 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:36.011520 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-add1bb-dockercfg-c5lhx\"" Oct 30 01:08:36.053859 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:36.053706 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls25m\" (UniqueName: \"kubernetes.io/projected/c0c76374-4140-42c2-9a0d-48a94a0e04b7-kube-api-access-ls25m\") pod \"payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw\" (UID: \"c0c76374-4140-42c2-9a0d-48a94a0e04b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw" Oct 30 01:08:36.054159 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:36.053984 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0c76374-4140-42c2-9a0d-48a94a0e04b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw\" (UID: \"c0c76374-4140-42c2-9a0d-48a94a0e04b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw" Oct 30 01:08:36.065010 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:36.064976 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4"] Oct 30 01:08:36.075392 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:36.074521 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4" Oct 30 01:08:36.148022 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:36.139551 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4"] Oct 30 01:08:36.173783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:36.173155 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8af3eb07-ad1d-4484-a586-5a3f71386369-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4\" (UID: \"8af3eb07-ad1d-4484-a586-5a3f71386369\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4" Oct 30 01:08:36.173783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:36.173253 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls25m\" (UniqueName: \"kubernetes.io/projected/c0c76374-4140-42c2-9a0d-48a94a0e04b7-kube-api-access-ls25m\") pod \"payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw\" (UID: \"c0c76374-4140-42c2-9a0d-48a94a0e04b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw" Oct 30 01:08:36.173783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:36.173466 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsfv9\" (UniqueName: \"kubernetes.io/projected/8af3eb07-ad1d-4484-a586-5a3f71386369-kube-api-access-bsfv9\") pod \"payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4\" (UID: \"8af3eb07-ad1d-4484-a586-5a3f71386369\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4" Oct 30 01:08:36.173783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:36.173532 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0c76374-4140-42c2-9a0d-48a94a0e04b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw\" (UID: \"c0c76374-4140-42c2-9a0d-48a94a0e04b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw" Oct 30 01:08:36.177781 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:36.174546 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0c76374-4140-42c2-9a0d-48a94a0e04b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw\" (UID: \"c0c76374-4140-42c2-9a0d-48a94a0e04b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw" Oct 30 01:08:36.241568 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:36.240995 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls25m\" (UniqueName: \"kubernetes.io/projected/c0c76374-4140-42c2-9a0d-48a94a0e04b7-kube-api-access-ls25m\") pod \"payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw\" (UID: \"c0c76374-4140-42c2-9a0d-48a94a0e04b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw" Oct 30 01:08:36.274855 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:36.274809 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsfv9\" (UniqueName: \"kubernetes.io/projected/8af3eb07-ad1d-4484-a586-5a3f71386369-kube-api-access-bsfv9\") pod \"payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4\" (UID: \"8af3eb07-ad1d-4484-a586-5a3f71386369\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4" Oct 30 01:08:36.275057 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:36.274982 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8af3eb07-ad1d-4484-a586-5a3f71386369-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4\" (UID: \"8af3eb07-ad1d-4484-a586-5a3f71386369\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4" Oct 30 01:08:36.276454 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:36.275890 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8af3eb07-ad1d-4484-a586-5a3f71386369-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4\" (UID: \"8af3eb07-ad1d-4484-a586-5a3f71386369\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4" Oct 30 01:08:36.314824 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:36.303437 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsfv9\" (UniqueName: \"kubernetes.io/projected/8af3eb07-ad1d-4484-a586-5a3f71386369-kube-api-access-bsfv9\") pod \"payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4\" (UID: \"8af3eb07-ad1d-4484-a586-5a3f71386369\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4" Oct 30 01:08:36.314824 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:36.304904 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw" Oct 30 01:08:36.501865 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:36.487076 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4" Oct 30 01:08:36.698617 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:36.698588 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw"] Oct 30 01:08:36.698748 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:08:36.698697 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c76374_4140_42c2_9a0d_48a94a0e04b7.slice/crio-2994c4ae93e18edc6c778f6296ba22bf14aa875a9e087bc4b1b979300901b664 WatchSource:0}: Error finding container 2994c4ae93e18edc6c778f6296ba22bf14aa875a9e087bc4b1b979300901b664: Status 404 returned error can't find the container with id 2994c4ae93e18edc6c778f6296ba22bf14aa875a9e087bc4b1b979300901b664 Oct 30 01:08:36.839834 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:36.839805 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4"] Oct 30 01:08:36.841746 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:08:36.841719 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8af3eb07_ad1d_4484_a586_5a3f71386369.slice/crio-b7bf8d837f6754b5003064e83ebe270f6efb939b4266f50b8ad70e4a918ebf62 WatchSource:0}: Error finding container b7bf8d837f6754b5003064e83ebe270f6efb939b4266f50b8ad70e4a918ebf62: Status 404 returned error can't find the container with id b7bf8d837f6754b5003064e83ebe270f6efb939b4266f50b8ad70e4a918ebf62 Oct 30 01:08:36.891224 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:36.891183 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:37.345855 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:37.345810 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4" event={"ID":"8af3eb07-ad1d-4484-a586-5a3f71386369","Type":"ContainerStarted","Data":"dc0aad0bb30f7ca076b32b0d380d8eb1468cbb3a400d7da1b61790b3b7cecfdc"} Oct 30 01:08:37.346541 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:37.346514 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4" event={"ID":"8af3eb07-ad1d-4484-a586-5a3f71386369","Type":"ContainerStarted","Data":"b7bf8d837f6754b5003064e83ebe270f6efb939b4266f50b8ad70e4a918ebf62"} Oct 30 01:08:37.348579 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:37.348554 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4" Oct 30 01:08:37.350612 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:37.350562 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw" event={"ID":"c0c76374-4140-42c2-9a0d-48a94a0e04b7","Type":"ContainerStarted","Data":"0b4e588f70888c3c55e3f54b6246afb20e4073590ae6c8f360eedf09dec0081b"} Oct 30 01:08:37.350851 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:37.350836 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw" event={"ID":"c0c76374-4140-42c2-9a0d-48a94a0e04b7","Type":"ContainerStarted","Data":"2994c4ae93e18edc6c778f6296ba22bf14aa875a9e087bc4b1b979300901b664"} Oct 30 01:08:37.352156 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:37.352119 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw" Oct 30 01:08:37.400784 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:37.398286 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4" podStartSLOduration=1.398263789 podStartE2EDuration="1.398263789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:37.394591832 +0000 UTC m=+1517.710460159" watchObservedRunningTime="2026-10-30 01:08:37.398263789 +0000 UTC m=+1517.714132115" Oct 30 01:08:37.442596 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:37.442533 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw" podStartSLOduration=2.442515546 podStartE2EDuration="2.442515546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:37.442247805 +0000 UTC m=+1517.758116164" watchObservedRunningTime="2026-10-30 01:08:37.442515546 +0000 UTC m=+1517.758383875" Oct 30 01:08:38.471127 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:38.471089 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:38.589921 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:38.589821 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:38.673066 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:38.673027 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:40.461570 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:40.461532 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:42.248134 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:42.248099 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:43.234044 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:43.233999 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:44.016404 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:44.016358 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:44.115024 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:44.114986 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:44.235446 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:44.235413 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:44.493274 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:44.493232 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:45.843696 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:45.843663 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:47.643108 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:47.643068 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:49.193257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:49.193219 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:49.369793 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:49.369732 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4" Oct 30 01:08:49.372683 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:49.372651 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw" Oct 30 01:08:50.887599 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:50.887502 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:51.595979 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:08:51.595943 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:08:52.737781 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:52.737737 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:54.184093 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:54.184058 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:54.523333 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:54.523297 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66"] Oct 30 01:08:54.523875 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:54.523840 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66" podUID="1e0387dc-9b10-4e10-b892-a027d042f83f" containerName="maas-api" containerID="cri-o://6716191b1a8a553e435ca9a79fd7f6dcc41f0b30ca83a5799fe4c2a223ccf0b7" gracePeriod=30 Oct 30 01:08:54.591050 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:54.589127 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:54.712808 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:54.712746 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:54.852845 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:54.845812 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:54.865372 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:54.865331 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66" Oct 30 01:08:54.921782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:54.917586 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e0387dc-9b10-4e10-b892-a027d042f83f-kube-api-access-cs6h8\" (UniqueName: \"kubernetes.io/projected/1e0387dc-9b10-4e10-b892-a027d042f83f-kube-api-access-cs6h8\") pod \"1e0387dc-9b10-4e10-b892-a027d042f83f\" (UID: \"1e0387dc-9b10-4e10-b892-a027d042f83f\") " Oct 30 01:08:54.921782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:54.917820 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e0387dc-9b10-4e10-b892-a027d042f83f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0387dc-9b10-4e10-b892-a027d042f83f-openshift-service-ca\") pod \"1e0387dc-9b10-4e10-b892-a027d042f83f\" (UID: \"1e0387dc-9b10-4e10-b892-a027d042f83f\") " Oct 30 01:08:54.921782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:54.917938 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e0387dc-9b10-4e10-b892-a027d042f83f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e0387dc-9b10-4e10-b892-a027d042f83f-maas-api-tls\") pod \"1e0387dc-9b10-4e10-b892-a027d042f83f\" (UID: \"1e0387dc-9b10-4e10-b892-a027d042f83f\") " Oct 30 01:08:54.921782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:54.918008 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e0387dc-9b10-4e10-b892-a027d042f83f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e0387dc-9b10-4e10-b892-a027d042f83f-metrics-certs\") pod \"1e0387dc-9b10-4e10-b892-a027d042f83f\" (UID: \"1e0387dc-9b10-4e10-b892-a027d042f83f\") " Oct 30 01:08:54.921782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:54.920024 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e0387dc-9b10-4e10-b892-a027d042f83f-openshift-service-ca" pod "1e0387dc-9b10-4e10-b892-a027d042f83f" (UID: "1e0387dc-9b10-4e10-b892-a027d042f83f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:08:54.931782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:54.928091 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e0387dc-9b10-4e10-b892-a027d042f83f-metrics-certs" pod "1e0387dc-9b10-4e10-b892-a027d042f83f" (UID: "1e0387dc-9b10-4e10-b892-a027d042f83f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:08:54.937819 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:54.934994 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e0387dc-9b10-4e10-b892-a027d042f83f-maas-api-tls" pod "1e0387dc-9b10-4e10-b892-a027d042f83f" (UID: "1e0387dc-9b10-4e10-b892-a027d042f83f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:08:54.949786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:54.948432 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0387dc-9b10-4e10-b892-a027d042f83f-kube-api-access-cs6h8" pod "1e0387dc-9b10-4e10-b892-a027d042f83f" (UID: "1e0387dc-9b10-4e10-b892-a027d042f83f"). InnerVolumeSpecName "kube-api-access-cs6h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:08:55.024703 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:55.024286 5300 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0387dc-9b10-4e10-b892-a027d042f83f-openshift-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:08:55.024703 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:55.024328 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e0387dc-9b10-4e10-b892-a027d042f83f-maas-api-tls\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:08:55.024703 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:55.024342 5300 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e0387dc-9b10-4e10-b892-a027d042f83f-metrics-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:08:55.024703 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:55.024356 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs6h8\" (UniqueName: \"kubernetes.io/projected/1e0387dc-9b10-4e10-b892-a027d042f83f-kube-api-access-cs6h8\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:08:55.262271 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:55.262236 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:55.418094 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:55.418060 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:55.500918 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:55.500879 5300 generic.go:356] "Generic (PLEG): container finished" podID="1e0387dc-9b10-4e10-b892-a027d042f83f" containerID="6716191b1a8a553e435ca9a79fd7f6dcc41f0b30ca83a5799fe4c2a223ccf0b7" exitCode=0 Oct 30 01:08:55.501140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:55.501122 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66" event={"ID":"1e0387dc-9b10-4e10-b892-a027d042f83f","Type":"ContainerDied","Data":"6716191b1a8a553e435ca9a79fd7f6dcc41f0b30ca83a5799fe4c2a223ccf0b7"} Oct 30 01:08:55.501216 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:55.501170 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66" event={"ID":"1e0387dc-9b10-4e10-b892-a027d042f83f","Type":"ContainerDied","Data":"18f08c25a027fc22a1898886dc4b825b5ffe7b248515306b8f6517db62665524"} Oct 30 01:08:55.501216 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:55.501194 5300 scope.go:122] "RemoveContainer" containerID="6716191b1a8a553e435ca9a79fd7f6dcc41f0b30ca83a5799fe4c2a223ccf0b7" Oct 30 01:08:55.501338 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:55.501326 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66" Oct 30 01:08:55.516127 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:55.516087 5300 scope.go:122] "RemoveContainer" containerID="6716191b1a8a553e435ca9a79fd7f6dcc41f0b30ca83a5799fe4c2a223ccf0b7" Oct 30 01:08:55.516636 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:08:55.516594 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6716191b1a8a553e435ca9a79fd7f6dcc41f0b30ca83a5799fe4c2a223ccf0b7\": container with ID starting with 6716191b1a8a553e435ca9a79fd7f6dcc41f0b30ca83a5799fe4c2a223ccf0b7 not found: ID does not exist" containerID="6716191b1a8a553e435ca9a79fd7f6dcc41f0b30ca83a5799fe4c2a223ccf0b7" Oct 30 01:08:55.516730 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:55.516638 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6716191b1a8a553e435ca9a79fd7f6dcc41f0b30ca83a5799fe4c2a223ccf0b7"} err="failed to get container status \"6716191b1a8a553e435ca9a79fd7f6dcc41f0b30ca83a5799fe4c2a223ccf0b7\": rpc error: code = NotFound desc = could not find container \"6716191b1a8a553e435ca9a79fd7f6dcc41f0b30ca83a5799fe4c2a223ccf0b7\": container with ID starting with 6716191b1a8a553e435ca9a79fd7f6dcc41f0b30ca83a5799fe4c2a223ccf0b7 not found: ID does not exist" Oct 30 01:08:55.555475 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:55.555420 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:55.560439 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:55.560406 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66"] Oct 30 01:08:55.573877 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:55.573840 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-260268-7b898b5f48-clq66"] Oct 30 01:08:55.825809 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:55.825674 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e0387dc-9b10-4e10-b892-a027d042f83f" path="/var/lib/kubelet/pods/1e0387dc-9b10-4e10-b892-a027d042f83f/volumes" Oct 30 01:08:55.852212 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:55.852175 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:57.229361 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:57.229322 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:08:59.155905 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:08:59.155861 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:00.357126 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:00.357073 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:00.542696 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:00.542660 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:01.000929 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:01.000830 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:02.131361 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:02.131322 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:02.232875 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:02.232815 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:02.472776 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:02.472693 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:02.745242 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:02.745147 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:04.839338 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:04.839294 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:06.502839 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:06.502783 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd"] Oct 30 01:09:06.503559 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:06.503542 5300 cpu_manager.go:414] "containerMap: removing container" podUID="1e0387dc-9b10-4e10-b892-a027d042f83f" containerName="maas-api" Oct 30 01:09:06.503621 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:06.503560 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e0387dc-9b10-4e10-b892-a027d042f83f" containerName="maas-api" Oct 30 01:09:06.503703 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:06.503693 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e0387dc-9b10-4e10-b892-a027d042f83f" containerName="maas-api" Oct 30 01:09:06.509096 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:06.509069 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd" Oct 30 01:09:06.527995 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:06.527960 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd"] Oct 30 01:09:06.530503 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:06.530472 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/127ad507-8fcc-4c89-be18-27f08b405f8d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd\" (UID: \"127ad507-8fcc-4c89-be18-27f08b405f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd" Oct 30 01:09:06.530689 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:06.530520 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/127ad507-8fcc-4c89-be18-27f08b405f8d-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd\" (UID: \"127ad507-8fcc-4c89-be18-27f08b405f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd" Oct 30 01:09:06.530803 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:06.530702 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k77xh\" (UniqueName: \"kubernetes.io/projected/127ad507-8fcc-4c89-be18-27f08b405f8d-kube-api-access-k77xh\") pod \"maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd\" (UID: \"127ad507-8fcc-4c89-be18-27f08b405f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd" Oct 30 01:09:06.530884 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:06.530828 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/127ad507-8fcc-4c89-be18-27f08b405f8d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd\" (UID: \"127ad507-8fcc-4c89-be18-27f08b405f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd" Oct 30 01:09:06.536256 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:06.536226 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-add1bb-serving-cert\"" Oct 30 01:09:06.582107 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:06.582075 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:06.632136 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:06.632092 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/127ad507-8fcc-4c89-be18-27f08b405f8d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd\" (UID: \"127ad507-8fcc-4c89-be18-27f08b405f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd" Oct 30 01:09:06.632351 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:09:06.632272 5300 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-serving-cert: secret "maas-api-e2e-shared-b-w1-add1bb-serving-cert" not found Oct 30 01:09:06.632351 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:06.632342 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/127ad507-8fcc-4c89-be18-27f08b405f8d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd\" (UID: \"127ad507-8fcc-4c89-be18-27f08b405f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd" Oct 30 01:09:06.632481 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:09:06.632386 5300 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/127ad507-8fcc-4c89-be18-27f08b405f8d-maas-api-tls podName:127ad507-8fcc-4c89-be18-27f08b405f8d nodeName:}" failed. No retries permitted until 2026-10-30 01:09:07.132363772 +0000 UTC m=+1547.448232092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/127ad507-8fcc-4c89-be18-27f08b405f8d-maas-api-tls") pod "maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd" (UID: "127ad507-8fcc-4c89-be18-27f08b405f8d") : secret "maas-api-e2e-shared-b-w1-add1bb-serving-cert" not found Oct 30 01:09:06.632481 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:06.632424 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/127ad507-8fcc-4c89-be18-27f08b405f8d-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd\" (UID: \"127ad507-8fcc-4c89-be18-27f08b405f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd" Oct 30 01:09:06.632595 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:06.632515 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k77xh\" (UniqueName: \"kubernetes.io/projected/127ad507-8fcc-4c89-be18-27f08b405f8d-kube-api-access-k77xh\") pod \"maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd\" (UID: \"127ad507-8fcc-4c89-be18-27f08b405f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd" Oct 30 01:09:06.633154 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:06.633129 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/127ad507-8fcc-4c89-be18-27f08b405f8d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd\" (UID: \"127ad507-8fcc-4c89-be18-27f08b405f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd" Oct 30 01:09:06.635084 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:06.635064 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/127ad507-8fcc-4c89-be18-27f08b405f8d-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd\" (UID: \"127ad507-8fcc-4c89-be18-27f08b405f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd" Oct 30 01:09:06.654265 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:06.654233 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k77xh\" (UniqueName: \"kubernetes.io/projected/127ad507-8fcc-4c89-be18-27f08b405f8d-kube-api-access-k77xh\") pod \"maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd\" (UID: \"127ad507-8fcc-4c89-be18-27f08b405f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd" Oct 30 01:09:06.679396 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:06.679357 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:06.937267 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:06.937181 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:07.140694 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:07.140652 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/127ad507-8fcc-4c89-be18-27f08b405f8d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd\" (UID: \"127ad507-8fcc-4c89-be18-27f08b405f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd" Oct 30 01:09:07.143563 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:07.143532 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/127ad507-8fcc-4c89-be18-27f08b405f8d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd\" (UID: \"127ad507-8fcc-4c89-be18-27f08b405f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd" Oct 30 01:09:07.426691 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:07.426638 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd" Oct 30 01:09:07.601305 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:07.601271 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:07.739077 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:07.739044 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd"] Oct 30 01:09:07.739384 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:09:07.739361 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod127ad507_8fcc_4c89_be18_27f08b405f8d.slice/crio-ce9fb5cc5877bf549f0e822571b00158c0cfbd1f5458dda865640b07bd90838e WatchSource:0}: Error finding container ce9fb5cc5877bf549f0e822571b00158c0cfbd1f5458dda865640b07bd90838e: Status 404 returned error can't find the container with id ce9fb5cc5877bf549f0e822571b00158c0cfbd1f5458dda865640b07bd90838e Oct 30 01:09:08.357231 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:08.357143 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:08.605641 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:08.605599 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd" event={"ID":"127ad507-8fcc-4c89-be18-27f08b405f8d","Type":"ContainerStarted","Data":"579f38bd1fffebcda0bb22c366c83b2150c2675762942929d16786a9bfd17799"} Oct 30 01:09:08.606128 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:08.605668 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd" event={"ID":"127ad507-8fcc-4c89-be18-27f08b405f8d","Type":"ContainerStarted","Data":"ce9fb5cc5877bf549f0e822571b00158c0cfbd1f5458dda865640b07bd90838e"} Oct 30 01:09:08.606128 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:08.605829 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd" Oct 30 01:09:08.656401 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:08.656282 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd" podStartSLOduration=2.3187396590000002 podStartE2EDuration="2.65626832s" totalImagesPullingTime="337.528661ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:08.652492404 +0000 UTC m=+1548.968360729" watchObservedRunningTime="2026-10-30 01:09:08.65626832 +0000 UTC m=+1548.972136645" Oct 30 01:09:10.208359 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:10.208325 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:11.959644 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:11.959610 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:13.550192 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:13.550155 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:13.652126 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:13.652088 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:13.751024 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:13.750988 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:14.619409 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:14.619379 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd" Oct 30 01:09:15.441792 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:15.441731 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:17.142813 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:17.142776 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:18.940142 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:18.940110 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:20.803211 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:20.803175 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:21.594608 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:09:21.594570 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:09:22.676878 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:22.676823 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:24.563031 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:24.562994 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:24.670305 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:24.666006 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:26.135717 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:26.135647 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:26.942232 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:26.942194 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:27.088862 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:27.088817 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:27.222958 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:27.222912 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:27.601115 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:27.601022 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:28.975323 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:28.975284 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:30.737459 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:30.737427 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:32.548698 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:32.548660 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:32.777795 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:32.777724 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:33.147582 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:33.147547 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:34.494190 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:34.494156 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:34.619152 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:34.619116 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:34.756039 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:34.755925 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:34.963604 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:34.963570 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:35.765924 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:35.765853 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:36.408915 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:36.408853 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:36.526234 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:36.526194 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:36.892962 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:36.892930 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:37.789038 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:37.789006 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw"] Oct 30 01:09:37.837590 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:37.837554 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg"] Oct 30 01:09:37.977415 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:37.977377 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:38.149712 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:38.149610 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:38.349130 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:38.349080 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:38.876159 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:38.876121 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:39.903731 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:39.903694 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:40.027860 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:40.027823 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:40.293404 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:40.293366 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:41.107066 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:41.107007 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:41.612597 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:41.612545 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:41.732308 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:41.732264 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:42.435172 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:42.435133 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:42.789608 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:42.789564 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw" podUID="b80a1557-9093-451b-aa8d-f088de9b2d61" containerName="payload-processing" containerID="cri-o://4d6a133638129c3ffc8c4ae69023104f0f31195abc47515c1fd09a8e12b0ae1d" gracePeriod=25 Oct 30 01:09:42.838205 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:42.838153 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg" podUID="58101d14-ecc3-456c-bb6a-e1472585152c" containerName="payload-pre-processing" containerID="cri-o://8f116f4ca1570d296f21f1850ccea1a35d20e3cead1bea6b7015c8f68707e7d6" gracePeriod=25 Oct 30 01:09:43.223989 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.223954 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg" Oct 30 01:09:43.234348 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.234317 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw" Oct 30 01:09:43.323620 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.323572 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58101d14-ecc3-456c-bb6a-e1472585152c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58101d14-ecc3-456c-bb6a-e1472585152c-plugins-config-volume\") pod \"58101d14-ecc3-456c-bb6a-e1472585152c\" (UID: \"58101d14-ecc3-456c-bb6a-e1472585152c\") " Oct 30 01:09:43.327789 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.326465 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b80a1557-9093-451b-aa8d-f088de9b2d61-kube-api-access-94t9f\" (UniqueName: \"kubernetes.io/projected/b80a1557-9093-451b-aa8d-f088de9b2d61-kube-api-access-94t9f\") pod \"b80a1557-9093-451b-aa8d-f088de9b2d61\" (UID: \"b80a1557-9093-451b-aa8d-f088de9b2d61\") " Oct 30 01:09:43.327789 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.326668 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58101d14-ecc3-456c-bb6a-e1472585152c-kube-api-access-bq77s\" (UniqueName: \"kubernetes.io/projected/58101d14-ecc3-456c-bb6a-e1472585152c-kube-api-access-bq77s\") pod \"58101d14-ecc3-456c-bb6a-e1472585152c\" (UID: \"58101d14-ecc3-456c-bb6a-e1472585152c\") " Oct 30 01:09:43.327789 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.326750 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b80a1557-9093-451b-aa8d-f088de9b2d61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b80a1557-9093-451b-aa8d-f088de9b2d61-plugins-config-volume\") pod \"b80a1557-9093-451b-aa8d-f088de9b2d61\" (UID: \"b80a1557-9093-451b-aa8d-f088de9b2d61\") " Oct 30 01:09:43.328563 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.326194 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58101d14-ecc3-456c-bb6a-e1472585152c-plugins-config-volume" pod "58101d14-ecc3-456c-bb6a-e1472585152c" (UID: "58101d14-ecc3-456c-bb6a-e1472585152c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:09:43.346191 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.346135 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b80a1557-9093-451b-aa8d-f088de9b2d61-plugins-config-volume" pod "b80a1557-9093-451b-aa8d-f088de9b2d61" (UID: "b80a1557-9093-451b-aa8d-f088de9b2d61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:09:43.395074 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.394978 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58101d14-ecc3-456c-bb6a-e1472585152c-kube-api-access-bq77s" pod "58101d14-ecc3-456c-bb6a-e1472585152c" (UID: "58101d14-ecc3-456c-bb6a-e1472585152c"). InnerVolumeSpecName "kube-api-access-bq77s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:43.395838 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.395755 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b80a1557-9093-451b-aa8d-f088de9b2d61-kube-api-access-94t9f" pod "b80a1557-9093-451b-aa8d-f088de9b2d61" (UID: "b80a1557-9093-451b-aa8d-f088de9b2d61"). InnerVolumeSpecName "kube-api-access-94t9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:43.430539 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.430499 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b80a1557-9093-451b-aa8d-f088de9b2d61-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:09:43.430539 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.430541 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58101d14-ecc3-456c-bb6a-e1472585152c-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:09:43.430806 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.430556 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94t9f\" (UniqueName: \"kubernetes.io/projected/b80a1557-9093-451b-aa8d-f088de9b2d61-kube-api-access-94t9f\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:09:43.430806 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.430571 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq77s\" (UniqueName: \"kubernetes.io/projected/58101d14-ecc3-456c-bb6a-e1472585152c-kube-api-access-bq77s\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:09:43.899666 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.899626 5300 generic.go:356] "Generic (PLEG): container finished" podID="58101d14-ecc3-456c-bb6a-e1472585152c" containerID="8f116f4ca1570d296f21f1850ccea1a35d20e3cead1bea6b7015c8f68707e7d6" exitCode=0 Oct 30 01:09:43.900175 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.899786 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg" Oct 30 01:09:43.900175 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.899825 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg" event={"ID":"58101d14-ecc3-456c-bb6a-e1472585152c","Type":"ContainerDied","Data":"8f116f4ca1570d296f21f1850ccea1a35d20e3cead1bea6b7015c8f68707e7d6"} Oct 30 01:09:43.900175 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.899896 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg" event={"ID":"58101d14-ecc3-456c-bb6a-e1472585152c","Type":"ContainerDied","Data":"faa3037970ebc4afc656cf1acbf494fee7a9d39980eba68830cfadb0362ae082"} Oct 30 01:09:43.900175 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.899933 5300 scope.go:122] "RemoveContainer" containerID="8f116f4ca1570d296f21f1850ccea1a35d20e3cead1bea6b7015c8f68707e7d6" Oct 30 01:09:43.903280 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.902122 5300 generic.go:356] "Generic (PLEG): container finished" podID="b80a1557-9093-451b-aa8d-f088de9b2d61" containerID="4d6a133638129c3ffc8c4ae69023104f0f31195abc47515c1fd09a8e12b0ae1d" exitCode=0 Oct 30 01:09:43.903280 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.902310 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw" event={"ID":"b80a1557-9093-451b-aa8d-f088de9b2d61","Type":"ContainerDied","Data":"4d6a133638129c3ffc8c4ae69023104f0f31195abc47515c1fd09a8e12b0ae1d"} Oct 30 01:09:43.903280 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.902363 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw" event={"ID":"b80a1557-9093-451b-aa8d-f088de9b2d61","Type":"ContainerDied","Data":"452a15336579a0f60a51345aa7989f444937c6515133ac5d89a77042107a726c"} Oct 30 01:09:43.903280 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.902440 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw" Oct 30 01:09:43.918582 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.918554 5300 scope.go:122] "RemoveContainer" containerID="8f116f4ca1570d296f21f1850ccea1a35d20e3cead1bea6b7015c8f68707e7d6" Oct 30 01:09:43.919083 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:09:43.919054 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f116f4ca1570d296f21f1850ccea1a35d20e3cead1bea6b7015c8f68707e7d6\": container with ID starting with 8f116f4ca1570d296f21f1850ccea1a35d20e3cead1bea6b7015c8f68707e7d6 not found: ID does not exist" containerID="8f116f4ca1570d296f21f1850ccea1a35d20e3cead1bea6b7015c8f68707e7d6" Oct 30 01:09:43.919206 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.919096 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f116f4ca1570d296f21f1850ccea1a35d20e3cead1bea6b7015c8f68707e7d6"} err="failed to get container status \"8f116f4ca1570d296f21f1850ccea1a35d20e3cead1bea6b7015c8f68707e7d6\": rpc error: code = NotFound desc = could not find container \"8f116f4ca1570d296f21f1850ccea1a35d20e3cead1bea6b7015c8f68707e7d6\": container with ID starting with 8f116f4ca1570d296f21f1850ccea1a35d20e3cead1bea6b7015c8f68707e7d6 not found: ID does not exist" Oct 30 01:09:43.919206 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.919119 5300 scope.go:122] "RemoveContainer" containerID="4d6a133638129c3ffc8c4ae69023104f0f31195abc47515c1fd09a8e12b0ae1d" Oct 30 01:09:43.932923 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.932895 5300 scope.go:122] "RemoveContainer" containerID="4d6a133638129c3ffc8c4ae69023104f0f31195abc47515c1fd09a8e12b0ae1d" Oct 30 01:09:43.933299 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:09:43.933274 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d6a133638129c3ffc8c4ae69023104f0f31195abc47515c1fd09a8e12b0ae1d\": container with ID starting with 4d6a133638129c3ffc8c4ae69023104f0f31195abc47515c1fd09a8e12b0ae1d not found: ID does not exist" containerID="4d6a133638129c3ffc8c4ae69023104f0f31195abc47515c1fd09a8e12b0ae1d" Oct 30 01:09:43.933387 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.933312 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d6a133638129c3ffc8c4ae69023104f0f31195abc47515c1fd09a8e12b0ae1d"} err="failed to get container status \"4d6a133638129c3ffc8c4ae69023104f0f31195abc47515c1fd09a8e12b0ae1d\": rpc error: code = NotFound desc = could not find container \"4d6a133638129c3ffc8c4ae69023104f0f31195abc47515c1fd09a8e12b0ae1d\": container with ID starting with 4d6a133638129c3ffc8c4ae69023104f0f31195abc47515c1fd09a8e12b0ae1d not found: ID does not exist" Oct 30 01:09:43.967961 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.967922 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw"] Oct 30 01:09:43.971724 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:43.971691 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-13a3e2-d4978667b-txkrw"] Oct 30 01:09:44.015635 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:44.015584 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg"] Oct 30 01:09:44.029824 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:44.018165 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-13a3e2-5c5445bb8c-6bhcg"] Oct 30 01:09:44.369866 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:44.368521 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:45.500528 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:45.500487 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:45.823589 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:45.823499 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58101d14-ecc3-456c-bb6a-e1472585152c" path="/var/lib/kubelet/pods/58101d14-ecc3-456c-bb6a-e1472585152c/volumes" Oct 30 01:09:45.824104 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:45.824086 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b80a1557-9093-451b-aa8d-f088de9b2d61" path="/var/lib/kubelet/pods/b80a1557-9093-451b-aa8d-f088de9b2d61/volumes" Oct 30 01:09:45.921573 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:45.921540 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:46.032038 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:46.032001 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:46.260107 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:46.260075 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:46.260688 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:46.260663 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6"] Oct 30 01:09:46.260978 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:46.260949 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" podUID="07ae9aeb-afa1-4307-a062-6e2ddb055395" containerName="istio-proxy" containerID="cri-o://420951f796b37a9cbe00156a839aa7e9a453842c93ca6164d7d731cedcc1bd84" gracePeriod=30 Oct 30 01:09:46.425841 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:46.421395 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:46.741494 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:46.741456 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:47.216311 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:47.216204 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:47.865490 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:47.865454 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:48.570734 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:48.570703 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:50.460259 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:50.460221 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:50.883652 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:50.883621 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:50.994901 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:50.994870 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:51.551824 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.551792 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:09:51.596976 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:09:51.596935 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:09:51.697004 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.696972 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-envoy\") pod \"07ae9aeb-afa1-4307-a062-6e2ddb055395\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " Oct 30 01:09:51.697187 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.697041 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07ae9aeb-afa1-4307-a062-6e2ddb055395-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07ae9aeb-afa1-4307-a062-6e2ddb055395-istiod-ca-cert\") pod \"07ae9aeb-afa1-4307-a062-6e2ddb055395\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " Oct 30 01:09:51.697187 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.697110 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-token\" (UniqueName: \"kubernetes.io/projected/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-token\") pod \"07ae9aeb-afa1-4307-a062-6e2ddb055395\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " Oct 30 01:09:51.697187 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.697153 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-credential-socket\") pod \"07ae9aeb-afa1-4307-a062-6e2ddb055395\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " Oct 30 01:09:51.697371 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.697281 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-data\") pod \"07ae9aeb-afa1-4307-a062-6e2ddb055395\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " Oct 30 01:09:51.697371 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.697310 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-podinfo\") pod \"07ae9aeb-afa1-4307-a062-6e2ddb055395\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " Oct 30 01:09:51.697371 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.697326 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-workload-certs\") pod \"07ae9aeb-afa1-4307-a062-6e2ddb055395\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " Oct 30 01:09:51.697527 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.697429 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07ae9aeb-afa1-4307-a062-6e2ddb055395-kube-api-access-x2vwn\" (UniqueName: \"kubernetes.io/projected/07ae9aeb-afa1-4307-a062-6e2ddb055395-kube-api-access-x2vwn\") pod \"07ae9aeb-afa1-4307-a062-6e2ddb055395\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " Oct 30 01:09:51.697527 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.697443 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-credential-socket" pod "07ae9aeb-afa1-4307-a062-6e2ddb055395" (UID: "07ae9aeb-afa1-4307-a062-6e2ddb055395"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:51.697643 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.697535 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-workload-socket\") pod \"07ae9aeb-afa1-4307-a062-6e2ddb055395\" (UID: \"07ae9aeb-afa1-4307-a062-6e2ddb055395\") " Oct 30 01:09:51.697643 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.697572 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-data" pod "07ae9aeb-afa1-4307-a062-6e2ddb055395" (UID: "07ae9aeb-afa1-4307-a062-6e2ddb055395"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:51.697643 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.697494 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07ae9aeb-afa1-4307-a062-6e2ddb055395-istiod-ca-cert" pod "07ae9aeb-afa1-4307-a062-6e2ddb055395" (UID: "07ae9aeb-afa1-4307-a062-6e2ddb055395"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:09:51.697912 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.697671 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-workload-certs" pod "07ae9aeb-afa1-4307-a062-6e2ddb055395" (UID: "07ae9aeb-afa1-4307-a062-6e2ddb055395"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:51.698005 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.697986 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-workload-socket" pod "07ae9aeb-afa1-4307-a062-6e2ddb055395" (UID: "07ae9aeb-afa1-4307-a062-6e2ddb055395"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:51.698304 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.698287 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07ae9aeb-afa1-4307-a062-6e2ddb055395-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:09:51.698389 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.698308 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:09:51.698389 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.698340 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:09:51.698389 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.698355 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:09:51.698389 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.698367 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:09:51.699602 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.699572 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-token" pod "07ae9aeb-afa1-4307-a062-6e2ddb055395" (UID: "07ae9aeb-afa1-4307-a062-6e2ddb055395"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:51.699710 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.699690 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-envoy" pod "07ae9aeb-afa1-4307-a062-6e2ddb055395" (UID: "07ae9aeb-afa1-4307-a062-6e2ddb055395"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:09:51.699710 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.699705 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ae9aeb-afa1-4307-a062-6e2ddb055395-kube-api-access-x2vwn" pod "07ae9aeb-afa1-4307-a062-6e2ddb055395" (UID: "07ae9aeb-afa1-4307-a062-6e2ddb055395"). InnerVolumeSpecName "kube-api-access-x2vwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:09:51.699821 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.699805 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-podinfo" pod "07ae9aeb-afa1-4307-a062-6e2ddb055395" (UID: "07ae9aeb-afa1-4307-a062-6e2ddb055395"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:09:51.799690 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.799585 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2vwn\" (UniqueName: \"kubernetes.io/projected/07ae9aeb-afa1-4307-a062-6e2ddb055395-kube-api-access-x2vwn\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:09:51.799690 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.799628 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:09:51.799690 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.799640 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:09:51.799690 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.799647 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07ae9aeb-afa1-4307-a062-6e2ddb055395-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:09:51.994971 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.994938 5300 generic.go:356] "Generic (PLEG): container finished" podID="07ae9aeb-afa1-4307-a062-6e2ddb055395" containerID="420951f796b37a9cbe00156a839aa7e9a453842c93ca6164d7d731cedcc1bd84" exitCode=0 Oct 30 01:09:51.995152 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.994998 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" Oct 30 01:09:51.995152 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.995024 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" event={"ID":"07ae9aeb-afa1-4307-a062-6e2ddb055395","Type":"ContainerDied","Data":"420951f796b37a9cbe00156a839aa7e9a453842c93ca6164d7d731cedcc1bd84"} Oct 30 01:09:51.995152 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.995088 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6" event={"ID":"07ae9aeb-afa1-4307-a062-6e2ddb055395","Type":"ContainerDied","Data":"daafc4b11973d686946fd920c72fb407f62ea1ad08bc2cbc935da70e405a96ac"} Oct 30 01:09:51.995152 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:51.995119 5300 scope.go:122] "RemoveContainer" containerID="420951f796b37a9cbe00156a839aa7e9a453842c93ca6164d7d731cedcc1bd84" Oct 30 01:09:52.005431 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:52.005412 5300 scope.go:122] "RemoveContainer" containerID="e428eed35e85bdb6820df1eb011e78408604865ea73b8515d5fa83ebe5ee6ec2" Oct 30 01:09:52.015127 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:52.015108 5300 scope.go:122] "RemoveContainer" containerID="420951f796b37a9cbe00156a839aa7e9a453842c93ca6164d7d731cedcc1bd84" Oct 30 01:09:52.015454 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:09:52.015421 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"420951f796b37a9cbe00156a839aa7e9a453842c93ca6164d7d731cedcc1bd84\": container with ID starting with 420951f796b37a9cbe00156a839aa7e9a453842c93ca6164d7d731cedcc1bd84 not found: ID does not exist" containerID="420951f796b37a9cbe00156a839aa7e9a453842c93ca6164d7d731cedcc1bd84" Oct 30 01:09:52.015535 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:52.015461 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"420951f796b37a9cbe00156a839aa7e9a453842c93ca6164d7d731cedcc1bd84"} err="failed to get container status \"420951f796b37a9cbe00156a839aa7e9a453842c93ca6164d7d731cedcc1bd84\": rpc error: code = NotFound desc = could not find container \"420951f796b37a9cbe00156a839aa7e9a453842c93ca6164d7d731cedcc1bd84\": container with ID starting with 420951f796b37a9cbe00156a839aa7e9a453842c93ca6164d7d731cedcc1bd84 not found: ID does not exist" Oct 30 01:09:52.015535 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:52.015480 5300 scope.go:122] "RemoveContainer" containerID="e428eed35e85bdb6820df1eb011e78408604865ea73b8515d5fa83ebe5ee6ec2" Oct 30 01:09:52.015783 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:09:52.015738 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e428eed35e85bdb6820df1eb011e78408604865ea73b8515d5fa83ebe5ee6ec2\": container with ID starting with e428eed35e85bdb6820df1eb011e78408604865ea73b8515d5fa83ebe5ee6ec2 not found: ID does not exist" containerID="e428eed35e85bdb6820df1eb011e78408604865ea73b8515d5fa83ebe5ee6ec2" Oct 30 01:09:52.015833 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:52.015792 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e428eed35e85bdb6820df1eb011e78408604865ea73b8515d5fa83ebe5ee6ec2"} err="failed to get container status \"e428eed35e85bdb6820df1eb011e78408604865ea73b8515d5fa83ebe5ee6ec2\": rpc error: code = NotFound desc = could not find container \"e428eed35e85bdb6820df1eb011e78408604865ea73b8515d5fa83ebe5ee6ec2\": container with ID starting with e428eed35e85bdb6820df1eb011e78408604865ea73b8515d5fa83ebe5ee6ec2 not found: ID does not exist" Oct 30 01:09:52.052569 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:52.052485 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6"] Oct 30 01:09:52.066348 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:52.066317 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-fb3370cb-openshift-default-7488cd865-zrth6"] Oct 30 01:09:52.378228 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:52.378133 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:52.645865 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:52.645786 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:52.742341 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:52.742306 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:53.822876 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:53.822840 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07ae9aeb-afa1-4307-a062-6e2ddb055395" path="/var/lib/kubelet/pods/07ae9aeb-afa1-4307-a062-6e2ddb055395/volumes" Oct 30 01:09:54.392884 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:54.392850 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:56.169065 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:56.169027 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:57.974822 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:57.974752 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:58.581032 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:58.580999 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:59.079053 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:59.079010 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:09:59.275916 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:09:59.275880 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:00.067586 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:00.067536 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:00.829706 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:00.829662 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:01.978329 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:01.978293 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:02.988512 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:02.988469 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:03.290719 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:03.290631 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:04.143200 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:04.143158 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:04.787940 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:04.787906 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:04.894710 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:04.894667 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:06.362046 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:06.362011 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:08.297342 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:08.297300 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:09.993552 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:09.993523 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:11.845539 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:11.845503 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:13.989070 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:13.989029 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:14.087355 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:14.087321 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:14.193008 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:14.192972 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:15.860001 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:15.859911 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:17.553092 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:17.553053 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:18.914257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:18.914217 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc"] Oct 30 01:10:18.914718 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:18.914437 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc" podUID="da3394c8-e374-44aa-ae07-590de2d9c4bf" containerName="maas-api" containerID="cri-o://1ee2c5f9ed483a68d76bfbf43de03d26cd1268aaa7e7b75ef336d643587979cc" gracePeriod=30 Oct 30 01:10:18.978173 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:18.978123 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:19.242363 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.242330 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc" Oct 30 01:10:19.369054 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.369003 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da3394c8-e374-44aa-ae07-590de2d9c4bf-kube-api-access-m4gdt\" (UniqueName: \"kubernetes.io/projected/da3394c8-e374-44aa-ae07-590de2d9c4bf-kube-api-access-m4gdt\") pod \"da3394c8-e374-44aa-ae07-590de2d9c4bf\" (UID: \"da3394c8-e374-44aa-ae07-590de2d9c4bf\") " Oct 30 01:10:19.369456 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.369424 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da3394c8-e374-44aa-ae07-590de2d9c4bf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3394c8-e374-44aa-ae07-590de2d9c4bf-openshift-service-ca\") pod \"da3394c8-e374-44aa-ae07-590de2d9c4bf\" (UID: \"da3394c8-e374-44aa-ae07-590de2d9c4bf\") " Oct 30 01:10:19.369644 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.369619 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da3394c8-e374-44aa-ae07-590de2d9c4bf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da3394c8-e374-44aa-ae07-590de2d9c4bf-metrics-certs\") pod \"da3394c8-e374-44aa-ae07-590de2d9c4bf\" (UID: \"da3394c8-e374-44aa-ae07-590de2d9c4bf\") " Oct 30 01:10:19.369781 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.369747 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da3394c8-e374-44aa-ae07-590de2d9c4bf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da3394c8-e374-44aa-ae07-590de2d9c4bf-maas-api-tls\") pod \"da3394c8-e374-44aa-ae07-590de2d9c4bf\" (UID: \"da3394c8-e374-44aa-ae07-590de2d9c4bf\") " Oct 30 01:10:19.372887 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.372861 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:19.373557 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.373533 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da3394c8-e374-44aa-ae07-590de2d9c4bf-maas-api-tls" pod "da3394c8-e374-44aa-ae07-590de2d9c4bf" (UID: "da3394c8-e374-44aa-ae07-590de2d9c4bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:10:19.374104 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.374075 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3394c8-e374-44aa-ae07-590de2d9c4bf-openshift-service-ca" pod "da3394c8-e374-44aa-ae07-590de2d9c4bf" (UID: "da3394c8-e374-44aa-ae07-590de2d9c4bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:10:19.388208 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.388147 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs"] Oct 30 01:10:19.388672 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.388638 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs" podUID="42434c3a-b364-4693-8493-c77ae7baf016" containerName="maas-api" containerID="cri-o://3ee0ef7a3b7da0e8db55c012a938215ab92af70ec981f672371fef06ad97ec30" gracePeriod=30 Oct 30 01:10:19.426861 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.426716 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da3394c8-e374-44aa-ae07-590de2d9c4bf-metrics-certs" pod "da3394c8-e374-44aa-ae07-590de2d9c4bf" (UID: "da3394c8-e374-44aa-ae07-590de2d9c4bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:10:19.431898 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.431865 5300 generic.go:356] "Generic (PLEG): container finished" podID="da3394c8-e374-44aa-ae07-590de2d9c4bf" containerID="1ee2c5f9ed483a68d76bfbf43de03d26cd1268aaa7e7b75ef336d643587979cc" exitCode=0 Oct 30 01:10:19.432242 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.432190 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc" event={"ID":"da3394c8-e374-44aa-ae07-590de2d9c4bf","Type":"ContainerDied","Data":"1ee2c5f9ed483a68d76bfbf43de03d26cd1268aaa7e7b75ef336d643587979cc"} Oct 30 01:10:19.432363 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.432278 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc" event={"ID":"da3394c8-e374-44aa-ae07-590de2d9c4bf","Type":"ContainerDied","Data":"c099689f4c5f5c7426cf035da6c740eec17271745adf53c460d697ea8e352c2a"} Oct 30 01:10:19.432363 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.432312 5300 scope.go:122] "RemoveContainer" containerID="1ee2c5f9ed483a68d76bfbf43de03d26cd1268aaa7e7b75ef336d643587979cc" Oct 30 01:10:19.432499 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.432481 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc" Oct 30 01:10:19.451222 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.451134 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3394c8-e374-44aa-ae07-590de2d9c4bf-kube-api-access-m4gdt" pod "da3394c8-e374-44aa-ae07-590de2d9c4bf" (UID: "da3394c8-e374-44aa-ae07-590de2d9c4bf"). InnerVolumeSpecName "kube-api-access-m4gdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:19.472499 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.472461 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4gdt\" (UniqueName: \"kubernetes.io/projected/da3394c8-e374-44aa-ae07-590de2d9c4bf-kube-api-access-m4gdt\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:10:19.472499 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.472501 5300 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3394c8-e374-44aa-ae07-590de2d9c4bf-openshift-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:10:19.472693 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.472519 5300 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da3394c8-e374-44aa-ae07-590de2d9c4bf-metrics-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:10:19.472693 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.472536 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da3394c8-e374-44aa-ae07-590de2d9c4bf-maas-api-tls\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:10:19.491055 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.491022 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:19.539172 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.538783 5300 scope.go:122] "RemoveContainer" containerID="1ee2c5f9ed483a68d76bfbf43de03d26cd1268aaa7e7b75ef336d643587979cc" Oct 30 01:10:19.539172 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:10:19.539132 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ee2c5f9ed483a68d76bfbf43de03d26cd1268aaa7e7b75ef336d643587979cc\": container with ID starting with 1ee2c5f9ed483a68d76bfbf43de03d26cd1268aaa7e7b75ef336d643587979cc not found: ID does not exist" containerID="1ee2c5f9ed483a68d76bfbf43de03d26cd1268aaa7e7b75ef336d643587979cc" Oct 30 01:10:19.539297 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.539173 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ee2c5f9ed483a68d76bfbf43de03d26cd1268aaa7e7b75ef336d643587979cc"} err="failed to get container status \"1ee2c5f9ed483a68d76bfbf43de03d26cd1268aaa7e7b75ef336d643587979cc\": rpc error: code = NotFound desc = could not find container \"1ee2c5f9ed483a68d76bfbf43de03d26cd1268aaa7e7b75ef336d643587979cc\": container with ID starting with 1ee2c5f9ed483a68d76bfbf43de03d26cd1268aaa7e7b75ef336d643587979cc not found: ID does not exist" Oct 30 01:10:19.605470 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.605430 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:19.775783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.755654 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs" Oct 30 01:10:19.893298 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.893251 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42434c3a-b364-4693-8493-c77ae7baf016-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42434c3a-b364-4693-8493-c77ae7baf016-maas-api-tls\") pod \"42434c3a-b364-4693-8493-c77ae7baf016\" (UID: \"42434c3a-b364-4693-8493-c77ae7baf016\") " Oct 30 01:10:19.893507 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.893489 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42434c3a-b364-4693-8493-c77ae7baf016-kube-api-access-72pgb\" (UniqueName: \"kubernetes.io/projected/42434c3a-b364-4693-8493-c77ae7baf016-kube-api-access-72pgb\") pod \"42434c3a-b364-4693-8493-c77ae7baf016\" (UID: \"42434c3a-b364-4693-8493-c77ae7baf016\") " Oct 30 01:10:19.893597 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.893536 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42434c3a-b364-4693-8493-c77ae7baf016-metrics-certs\" (UniqueName: \"kubernetes.io/secret/42434c3a-b364-4693-8493-c77ae7baf016-metrics-certs\") pod \"42434c3a-b364-4693-8493-c77ae7baf016\" (UID: \"42434c3a-b364-4693-8493-c77ae7baf016\") " Oct 30 01:10:19.894038 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.893629 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42434c3a-b364-4693-8493-c77ae7baf016-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42434c3a-b364-4693-8493-c77ae7baf016-openshift-service-ca\") pod \"42434c3a-b364-4693-8493-c77ae7baf016\" (UID: \"42434c3a-b364-4693-8493-c77ae7baf016\") " Oct 30 01:10:19.895005 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.894973 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42434c3a-b364-4693-8493-c77ae7baf016-openshift-service-ca" pod "42434c3a-b364-4693-8493-c77ae7baf016" (UID: "42434c3a-b364-4693-8493-c77ae7baf016"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:10:19.901713 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.901178 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc"] Oct 30 01:10:19.901713 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.901240 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-07410c-685fb97884-wfzqc"] Oct 30 01:10:19.903455 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.903415 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42434c3a-b364-4693-8493-c77ae7baf016-kube-api-access-72pgb" pod "42434c3a-b364-4693-8493-c77ae7baf016" (UID: "42434c3a-b364-4693-8493-c77ae7baf016"). InnerVolumeSpecName "kube-api-access-72pgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:19.904245 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.904199 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42434c3a-b364-4693-8493-c77ae7baf016-maas-api-tls" pod "42434c3a-b364-4693-8493-c77ae7baf016" (UID: "42434c3a-b364-4693-8493-c77ae7baf016"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:10:19.905725 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.905684 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42434c3a-b364-4693-8493-c77ae7baf016-metrics-certs" pod "42434c3a-b364-4693-8493-c77ae7baf016" (UID: "42434c3a-b364-4693-8493-c77ae7baf016"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:10:19.995744 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.995708 5300 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42434c3a-b364-4693-8493-c77ae7baf016-openshift-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:10:19.995744 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.995741 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42434c3a-b364-4693-8493-c77ae7baf016-maas-api-tls\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:10:19.996341 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.995772 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72pgb\" (UniqueName: \"kubernetes.io/projected/42434c3a-b364-4693-8493-c77ae7baf016-kube-api-access-72pgb\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:10:19.996341 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:19.995781 5300 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42434c3a-b364-4693-8493-c77ae7baf016-metrics-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:10:20.263053 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:20.262929 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:20.360987 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:20.360932 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:20.496786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:20.486106 5300 generic.go:356] "Generic (PLEG): container finished" podID="42434c3a-b364-4693-8493-c77ae7baf016" containerID="3ee0ef7a3b7da0e8db55c012a938215ab92af70ec981f672371fef06ad97ec30" exitCode=0 Oct 30 01:10:20.496786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:20.486169 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs" event={"ID":"42434c3a-b364-4693-8493-c77ae7baf016","Type":"ContainerDied","Data":"3ee0ef7a3b7da0e8db55c012a938215ab92af70ec981f672371fef06ad97ec30"} Oct 30 01:10:20.496786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:20.486315 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs" Oct 30 01:10:20.496786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:20.486603 5300 scope.go:122] "RemoveContainer" containerID="3ee0ef7a3b7da0e8db55c012a938215ab92af70ec981f672371fef06ad97ec30" Oct 30 01:10:20.514129 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:20.486494 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs" event={"ID":"42434c3a-b364-4693-8493-c77ae7baf016","Type":"ContainerDied","Data":"3838321a78f329d6a376ad04c38a66dd95fbc1ab63bfe9ed083486921ce2d67f"} Oct 30 01:10:20.514129 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:20.506320 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:20.712255 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:20.710735 5300 scope.go:122] "RemoveContainer" containerID="3ee0ef7a3b7da0e8db55c012a938215ab92af70ec981f672371fef06ad97ec30" Oct 30 01:10:20.712255 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:20.711188 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs"] Oct 30 01:10:20.712255 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:10:20.711564 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ee0ef7a3b7da0e8db55c012a938215ab92af70ec981f672371fef06ad97ec30\": container with ID starting with 3ee0ef7a3b7da0e8db55c012a938215ab92af70ec981f672371fef06ad97ec30 not found: ID does not exist" containerID="3ee0ef7a3b7da0e8db55c012a938215ab92af70ec981f672371fef06ad97ec30" Oct 30 01:10:20.712255 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:20.711605 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ee0ef7a3b7da0e8db55c012a938215ab92af70ec981f672371fef06ad97ec30"} err="failed to get container status \"3ee0ef7a3b7da0e8db55c012a938215ab92af70ec981f672371fef06ad97ec30\": rpc error: code = NotFound desc = could not find container \"3ee0ef7a3b7da0e8db55c012a938215ab92af70ec981f672371fef06ad97ec30\": container with ID starting with 3ee0ef7a3b7da0e8db55c012a938215ab92af70ec981f672371fef06ad97ec30 not found: ID does not exist" Oct 30 01:10:20.716643 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:20.716614 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-13a3e2-7c98c466c5-4gzjs"] Oct 30 01:10:20.950851 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:20.949036 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:21.704812 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:10:21.702569 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:10:22.011497 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:22.011465 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42434c3a-b364-4693-8493-c77ae7baf016" path="/var/lib/kubelet/pods/42434c3a-b364-4693-8493-c77ae7baf016/volumes" Oct 30 01:10:22.011997 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:22.011980 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da3394c8-e374-44aa-ae07-590de2d9c4bf" path="/var/lib/kubelet/pods/da3394c8-e374-44aa-ae07-590de2d9c4bf/volumes" Oct 30 01:10:22.631952 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:22.631908 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:24.450466 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:24.450432 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:26.229893 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.229860 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq"] Oct 30 01:10:26.230732 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.230709 5300 cpu_manager.go:414] "containerMap: removing container" podUID="07ae9aeb-afa1-4307-a062-6e2ddb055395" containerName="istio-proxy" Oct 30 01:10:26.230862 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.230733 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ae9aeb-afa1-4307-a062-6e2ddb055395" containerName="istio-proxy" Oct 30 01:10:26.230862 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.230774 5300 cpu_manager.go:414] "containerMap: removing container" podUID="07ae9aeb-afa1-4307-a062-6e2ddb055395" containerName="istio-proxy" Oct 30 01:10:26.230862 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.230782 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ae9aeb-afa1-4307-a062-6e2ddb055395" containerName="istio-proxy" Oct 30 01:10:26.230862 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.230811 5300 cpu_manager.go:414] "containerMap: removing container" podUID="b80a1557-9093-451b-aa8d-f088de9b2d61" containerName="payload-processing" Oct 30 01:10:26.230862 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.230819 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b80a1557-9093-451b-aa8d-f088de9b2d61" containerName="payload-processing" Oct 30 01:10:26.230862 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.230845 5300 cpu_manager.go:414] "containerMap: removing container" podUID="42434c3a-b364-4693-8493-c77ae7baf016" containerName="maas-api" Oct 30 01:10:26.230862 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.230852 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42434c3a-b364-4693-8493-c77ae7baf016" containerName="maas-api" Oct 30 01:10:26.231204 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.230867 5300 cpu_manager.go:414] "containerMap: removing container" podUID="da3394c8-e374-44aa-ae07-590de2d9c4bf" containerName="maas-api" Oct 30 01:10:26.231204 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.230874 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da3394c8-e374-44aa-ae07-590de2d9c4bf" containerName="maas-api" Oct 30 01:10:26.231204 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.230912 5300 cpu_manager.go:414] "containerMap: removing container" podUID="58101d14-ecc3-456c-bb6a-e1472585152c" containerName="payload-pre-processing" Oct 30 01:10:26.231204 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.230920 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58101d14-ecc3-456c-bb6a-e1472585152c" containerName="payload-pre-processing" Oct 30 01:10:26.231204 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.231113 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="58101d14-ecc3-456c-bb6a-e1472585152c" containerName="payload-pre-processing" Oct 30 01:10:26.231204 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.231129 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="07ae9aeb-afa1-4307-a062-6e2ddb055395" containerName="istio-proxy" Oct 30 01:10:26.231204 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.231145 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="b80a1557-9093-451b-aa8d-f088de9b2d61" containerName="payload-processing" Oct 30 01:10:26.231204 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.231158 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="42434c3a-b364-4693-8493-c77ae7baf016" containerName="maas-api" Oct 30 01:10:26.231204 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.231167 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="da3394c8-e374-44aa-ae07-590de2d9c4bf" containerName="maas-api" Oct 30 01:10:26.236463 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.236437 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq" Oct 30 01:10:26.253718 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.253683 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq"] Oct 30 01:10:26.264003 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.263970 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e92dab-serving-cert\"" Oct 30 01:10:26.316726 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.316676 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:26.375684 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.375633 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rr6r\" (UniqueName: \"kubernetes.io/projected/00621ae7-7e53-415c-a3a7-174a79440f4d-kube-api-access-7rr6r\") pod \"maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq\" (UID: \"00621ae7-7e53-415c-a3a7-174a79440f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq" Oct 30 01:10:26.375997 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.375916 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00621ae7-7e53-415c-a3a7-174a79440f4d-metrics-certs\") pod \"maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq\" (UID: \"00621ae7-7e53-415c-a3a7-174a79440f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq" Oct 30 01:10:26.376063 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.376036 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00621ae7-7e53-415c-a3a7-174a79440f4d-maas-api-tls\") pod \"maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq\" (UID: \"00621ae7-7e53-415c-a3a7-174a79440f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq" Oct 30 01:10:26.376299 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.376279 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00621ae7-7e53-415c-a3a7-174a79440f4d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq\" (UID: \"00621ae7-7e53-415c-a3a7-174a79440f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq" Oct 30 01:10:26.429574 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.429498 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:26.477808 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.477740 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00621ae7-7e53-415c-a3a7-174a79440f4d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq\" (UID: \"00621ae7-7e53-415c-a3a7-174a79440f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq" Oct 30 01:10:26.478040 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.477862 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rr6r\" (UniqueName: \"kubernetes.io/projected/00621ae7-7e53-415c-a3a7-174a79440f4d-kube-api-access-7rr6r\") pod \"maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq\" (UID: \"00621ae7-7e53-415c-a3a7-174a79440f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq" Oct 30 01:10:26.478040 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.477970 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00621ae7-7e53-415c-a3a7-174a79440f4d-metrics-certs\") pod \"maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq\" (UID: \"00621ae7-7e53-415c-a3a7-174a79440f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq" Oct 30 01:10:26.478040 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.478029 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00621ae7-7e53-415c-a3a7-174a79440f4d-maas-api-tls\") pod \"maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq\" (UID: \"00621ae7-7e53-415c-a3a7-174a79440f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq" Oct 30 01:10:26.478630 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.478600 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00621ae7-7e53-415c-a3a7-174a79440f4d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq\" (UID: \"00621ae7-7e53-415c-a3a7-174a79440f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq" Oct 30 01:10:26.480894 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.480789 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00621ae7-7e53-415c-a3a7-174a79440f4d-metrics-certs\") pod \"maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq\" (UID: \"00621ae7-7e53-415c-a3a7-174a79440f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq" Oct 30 01:10:26.480894 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.480835 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00621ae7-7e53-415c-a3a7-174a79440f4d-maas-api-tls\") pod \"maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq\" (UID: \"00621ae7-7e53-415c-a3a7-174a79440f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq" Oct 30 01:10:26.505742 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.505703 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rr6r\" (UniqueName: \"kubernetes.io/projected/00621ae7-7e53-415c-a3a7-174a79440f4d-kube-api-access-7rr6r\") pod \"maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq\" (UID: \"00621ae7-7e53-415c-a3a7-174a79440f4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq" Oct 30 01:10:26.551638 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.551599 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq" Oct 30 01:10:26.668052 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.668016 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:26.859671 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:26.859638 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq"] Oct 30 01:10:26.859942 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:10:26.859918 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00621ae7_7e53_415c_a3a7_174a79440f4d.slice/crio-41e02190b5aee359639a746b52c8be38fea85b5a9ce9f709f752d23360af1f90 WatchSource:0}: Error finding container 41e02190b5aee359639a746b52c8be38fea85b5a9ce9f709f752d23360af1f90: Status 404 returned error can't find the container with id 41e02190b5aee359639a746b52c8be38fea85b5a9ce9f709f752d23360af1f90 Oct 30 01:10:27.211573 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:27.211536 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:27.317832 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:27.317731 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:27.551705 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:27.551664 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:27.701852 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:27.699078 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq" event={"ID":"00621ae7-7e53-415c-a3a7-174a79440f4d","Type":"ContainerStarted","Data":"3c647ceee31f932063cce742e9206ca57283ef77e890c66367dafd1bea13ee55"} Oct 30 01:10:27.701852 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:27.699192 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq" event={"ID":"00621ae7-7e53-415c-a3a7-174a79440f4d","Type":"ContainerStarted","Data":"41e02190b5aee359639a746b52c8be38fea85b5a9ce9f709f752d23360af1f90"} Oct 30 01:10:27.701852 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:27.700529 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq" Oct 30 01:10:27.767171 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:27.767109 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq" podStartSLOduration=1.434562866 podStartE2EDuration="1.767090711s" totalImagesPullingTime="332.527845ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:27.765368529 +0000 UTC m=+1628.081236855" watchObservedRunningTime="2026-10-30 01:10:27.767090711 +0000 UTC m=+1628.082959038" Oct 30 01:10:28.181567 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:28.181519 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:28.354643 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:28.354608 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:28.948314 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:28.948273 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:29.047212 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:29.047181 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:29.237838 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:29.237752 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:29.343892 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:29.343853 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw"] Oct 30 01:10:29.344161 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:29.344127 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" podUID="c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2" containerName="istio-proxy" containerID="cri-o://7aec454b3ac8709091367196e5282c60b7d4d18d97b3b9e4b44306297e302743" gracePeriod=30 Oct 30 01:10:29.358927 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:29.358893 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:30.110657 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:30.110621 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:32.309605 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:32.309493 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:32.654407 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:32.654325 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:33.816467 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:33.816424 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:33.946983 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:33.946946 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:34.113972 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.113863 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:34.321616 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.321581 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:34.662071 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.662041 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:10:34.720977 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.720949 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq" Oct 30 01:10:34.732256 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.732217 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-podinfo\") pod \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " Oct 30 01:10:34.732464 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.732292 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istiod-ca-cert\") pod \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " Oct 30 01:10:34.732464 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.732364 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-kube-api-access-th2bz\" (UniqueName: \"kubernetes.io/projected/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-kube-api-access-th2bz\") pod \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " Oct 30 01:10:34.732464 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.732428 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-data\") pod \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " Oct 30 01:10:34.732624 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.732497 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-credential-socket\") pod \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " Oct 30 01:10:34.732684 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.732619 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-token\" (UniqueName: \"kubernetes.io/projected/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-token\") pod \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " Oct 30 01:10:34.732684 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.732671 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-workload-certs\") pod \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " Oct 30 01:10:34.732826 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.732752 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-envoy\") pod \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " Oct 30 01:10:34.732899 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.732878 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-workload-socket\") pod \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\" (UID: \"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2\") " Oct 30 01:10:34.733938 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.733901 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-workload-socket" pod "c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2" (UID: "c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:34.734891 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.734862 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-workload-certs" pod "c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2" (UID: "c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:34.735210 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.735171 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-data" pod "c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2" (UID: "c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:34.735368 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.735334 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istiod-ca-cert" pod "c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2" (UID: "c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:10:34.735506 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.735486 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-credential-socket" pod "c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2" (UID: "c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:34.750519 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.746691 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-podinfo" pod "c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2" (UID: "c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:10:34.761393 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.761355 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-kube-api-access-th2bz" pod "c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2" (UID: "c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2"). InnerVolumeSpecName "kube-api-access-th2bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:34.765475 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.765435 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-envoy" pod "c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2" (UID: "c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:10:34.769950 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.769913 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-token" pod "c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2" (UID: "c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:34.776214 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.776177 5300 generic.go:356] "Generic (PLEG): container finished" podID="c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2" containerID="7aec454b3ac8709091367196e5282c60b7d4d18d97b3b9e4b44306297e302743" exitCode=0 Oct 30 01:10:34.776362 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.776303 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" event={"ID":"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2","Type":"ContainerDied","Data":"7aec454b3ac8709091367196e5282c60b7d4d18d97b3b9e4b44306297e302743"} Oct 30 01:10:34.776434 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.776369 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" event={"ID":"c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2","Type":"ContainerDied","Data":"d553131254a01a98a31e57bd99fca0d583dffadf07e025436492b12a7b286076"} Oct 30 01:10:34.776434 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.776390 5300 scope.go:122] "RemoveContainer" containerID="7aec454b3ac8709091367196e5282c60b7d4d18d97b3b9e4b44306297e302743" Oct 30 01:10:34.776539 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.776524 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw" Oct 30 01:10:34.796195 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.796163 5300 scope.go:122] "RemoveContainer" containerID="f916fe3c48aaad148fdacf849fa41dc4e50ea86dc4e2d80e52712f4f047d6ce4" Oct 30 01:10:34.809743 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.809706 5300 scope.go:122] "RemoveContainer" containerID="7aec454b3ac8709091367196e5282c60b7d4d18d97b3b9e4b44306297e302743" Oct 30 01:10:34.810822 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:10:34.810785 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aec454b3ac8709091367196e5282c60b7d4d18d97b3b9e4b44306297e302743\": container with ID starting with 7aec454b3ac8709091367196e5282c60b7d4d18d97b3b9e4b44306297e302743 not found: ID does not exist" containerID="7aec454b3ac8709091367196e5282c60b7d4d18d97b3b9e4b44306297e302743" Oct 30 01:10:34.810960 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.810836 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aec454b3ac8709091367196e5282c60b7d4d18d97b3b9e4b44306297e302743"} err="failed to get container status \"7aec454b3ac8709091367196e5282c60b7d4d18d97b3b9e4b44306297e302743\": rpc error: code = NotFound desc = could not find container \"7aec454b3ac8709091367196e5282c60b7d4d18d97b3b9e4b44306297e302743\": container with ID starting with 7aec454b3ac8709091367196e5282c60b7d4d18d97b3b9e4b44306297e302743 not found: ID does not exist" Oct 30 01:10:34.810960 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.810864 5300 scope.go:122] "RemoveContainer" containerID="f916fe3c48aaad148fdacf849fa41dc4e50ea86dc4e2d80e52712f4f047d6ce4" Oct 30 01:10:34.811235 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:10:34.811207 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f916fe3c48aaad148fdacf849fa41dc4e50ea86dc4e2d80e52712f4f047d6ce4\": container with ID starting with f916fe3c48aaad148fdacf849fa41dc4e50ea86dc4e2d80e52712f4f047d6ce4 not found: ID does not exist" containerID="f916fe3c48aaad148fdacf849fa41dc4e50ea86dc4e2d80e52712f4f047d6ce4" Oct 30 01:10:34.811369 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.811244 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f916fe3c48aaad148fdacf849fa41dc4e50ea86dc4e2d80e52712f4f047d6ce4"} err="failed to get container status \"f916fe3c48aaad148fdacf849fa41dc4e50ea86dc4e2d80e52712f4f047d6ce4\": rpc error: code = NotFound desc = could not find container \"f916fe3c48aaad148fdacf849fa41dc4e50ea86dc4e2d80e52712f4f047d6ce4\": container with ID starting with f916fe3c48aaad148fdacf849fa41dc4e50ea86dc4e2d80e52712f4f047d6ce4 not found: ID does not exist" Oct 30 01:10:34.845125 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.845089 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th2bz\" (UniqueName: \"kubernetes.io/projected/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-kube-api-access-th2bz\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:10:34.845125 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.845124 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:10:34.845656 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.845140 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:10:34.845656 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.845153 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:10:34.845656 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.845168 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:10:34.845656 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.845180 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:10:34.845656 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.845194 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:10:34.845656 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.845207 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:10:34.845656 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.845222 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:10:34.851031 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.848101 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw"] Oct 30 01:10:34.853012 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:34.852985 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-13a3e2-openshift-default-5868ff964c-vbcvw"] Oct 30 01:10:35.796520 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:35.796488 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:35.823302 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:35.823264 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2" path="/var/lib/kubelet/pods/c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2/volumes" Oct 30 01:10:35.897352 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:35.897321 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:35.989219 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:35.989180 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9"] Oct 30 01:10:35.989956 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:35.989938 5300 cpu_manager.go:414] "containerMap: removing container" podUID="c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2" containerName="istio-proxy" Oct 30 01:10:35.989956 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:35.989956 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2" containerName="istio-proxy" Oct 30 01:10:35.990096 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:35.990077 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2" containerName="istio-proxy" Oct 30 01:10:35.990139 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:35.990100 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="07ae9aeb-afa1-4307-a062-6e2ddb055395" containerName="istio-proxy" Oct 30 01:10:35.990139 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:35.990111 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2" containerName="istio-proxy" Oct 30 01:10:35.995619 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:35.995592 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9" Oct 30 01:10:36.030433 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:36.030392 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4dde48\"" Oct 30 01:10:36.035141 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:36.035107 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9"] Oct 30 01:10:36.049895 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:36.044599 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4dde48-dockercfg-dz5zg\"" Oct 30 01:10:36.064408 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:36.064362 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959acd3d-9ea2-49bb-b791-5b87e28d693b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9\" (UID: \"959acd3d-9ea2-49bb-b791-5b87e28d693b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9" Oct 30 01:10:36.064725 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:36.064701 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v875\" (UniqueName: \"kubernetes.io/projected/959acd3d-9ea2-49bb-b791-5b87e28d693b-kube-api-access-7v875\") pod \"payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9\" (UID: \"959acd3d-9ea2-49bb-b791-5b87e28d693b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9" Oct 30 01:10:36.172787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:36.163560 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2"] Oct 30 01:10:36.172787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:36.164676 5300 cpu_manager.go:414] "containerMap: removing container" podUID="c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2" containerName="istio-proxy" Oct 30 01:10:36.172787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:36.164702 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7d8ad3c-1ff3-4331-af98-47ffa5aff2f2" containerName="istio-proxy" Oct 30 01:10:36.173853 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:36.173591 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2" Oct 30 01:10:36.183319 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:36.183284 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2"] Oct 30 01:10:36.189782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:36.189011 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959acd3d-9ea2-49bb-b791-5b87e28d693b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9\" (UID: \"959acd3d-9ea2-49bb-b791-5b87e28d693b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9" Oct 30 01:10:36.189782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:36.189337 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v875\" (UniqueName: \"kubernetes.io/projected/959acd3d-9ea2-49bb-b791-5b87e28d693b-kube-api-access-7v875\") pod \"payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9\" (UID: \"959acd3d-9ea2-49bb-b791-5b87e28d693b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9" Oct 30 01:10:36.191005 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:36.190736 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959acd3d-9ea2-49bb-b791-5b87e28d693b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9\" (UID: \"959acd3d-9ea2-49bb-b791-5b87e28d693b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9" Oct 30 01:10:36.319704 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:36.315214 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:36.339801 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:36.323383 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57510f64-4e51-4f32-bfbb-c9730bdb81c3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2\" (UID: \"57510f64-4e51-4f32-bfbb-c9730bdb81c3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2" Oct 30 01:10:36.340832 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:36.340799 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bntmh\" (UniqueName: \"kubernetes.io/projected/57510f64-4e51-4f32-bfbb-c9730bdb81c3-kube-api-access-bntmh\") pod \"payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2\" (UID: \"57510f64-4e51-4f32-bfbb-c9730bdb81c3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2" Oct 30 01:10:36.380720 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:36.380671 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v875\" (UniqueName: \"kubernetes.io/projected/959acd3d-9ea2-49bb-b791-5b87e28d693b-kube-api-access-7v875\") pod \"payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9\" (UID: \"959acd3d-9ea2-49bb-b791-5b87e28d693b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9" Oct 30 01:10:36.456142 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:36.456103 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bntmh\" (UniqueName: \"kubernetes.io/projected/57510f64-4e51-4f32-bfbb-c9730bdb81c3-kube-api-access-bntmh\") pod \"payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2\" (UID: \"57510f64-4e51-4f32-bfbb-c9730bdb81c3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2" Oct 30 01:10:36.456339 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:36.456300 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57510f64-4e51-4f32-bfbb-c9730bdb81c3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2\" (UID: \"57510f64-4e51-4f32-bfbb-c9730bdb81c3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2" Oct 30 01:10:36.457138 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:36.457110 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57510f64-4e51-4f32-bfbb-c9730bdb81c3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2\" (UID: \"57510f64-4e51-4f32-bfbb-c9730bdb81c3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2" Oct 30 01:10:36.477350 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:36.477310 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bntmh\" (UniqueName: \"kubernetes.io/projected/57510f64-4e51-4f32-bfbb-c9730bdb81c3-kube-api-access-bntmh\") pod \"payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2\" (UID: \"57510f64-4e51-4f32-bfbb-c9730bdb81c3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2" Oct 30 01:10:36.549829 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:36.549781 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2" Oct 30 01:10:36.611970 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:36.611888 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9" Oct 30 01:10:36.861419 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:36.861390 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2"] Oct 30 01:10:36.862405 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:10:36.862339 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57510f64_4e51_4f32_bfbb_c9730bdb81c3.slice/crio-52e1d85707d235ef4513ee015f958e0889fc2fcf8712c789cce96ce63a67074e WatchSource:0}: Error finding container 52e1d85707d235ef4513ee015f958e0889fc2fcf8712c789cce96ce63a67074e: Status 404 returned error can't find the container with id 52e1d85707d235ef4513ee015f958e0889fc2fcf8712c789cce96ce63a67074e Oct 30 01:10:36.949753 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:36.949723 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9"] Oct 30 01:10:36.950676 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:10:36.950651 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod959acd3d_9ea2_49bb_b791_5b87e28d693b.slice/crio-e71754692a8d2e921e65f6b4b754ebbc7623bb12d83a060b0c884434190f54b7 WatchSource:0}: Error finding container e71754692a8d2e921e65f6b4b754ebbc7623bb12d83a060b0c884434190f54b7: Status 404 returned error can't find the container with id e71754692a8d2e921e65f6b4b754ebbc7623bb12d83a060b0c884434190f54b7 Oct 30 01:10:37.803673 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:37.803638 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2" event={"ID":"57510f64-4e51-4f32-bfbb-c9730bdb81c3","Type":"ContainerStarted","Data":"70e387913b828fc7779a2ec0809ad3fc2f306621810af2501929bc2024d94489"} Oct 30 01:10:37.803899 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:37.803694 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2" event={"ID":"57510f64-4e51-4f32-bfbb-c9730bdb81c3","Type":"ContainerStarted","Data":"52e1d85707d235ef4513ee015f958e0889fc2fcf8712c789cce96ce63a67074e"} Oct 30 01:10:37.803899 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:37.803753 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2" Oct 30 01:10:37.805499 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:37.805473 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9" event={"ID":"959acd3d-9ea2-49bb-b791-5b87e28d693b","Type":"ContainerStarted","Data":"936a482b38fd7392f484ba736859c1433864dc2d049b14d8742bf9c571f2530e"} Oct 30 01:10:37.805605 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:37.805522 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9" event={"ID":"959acd3d-9ea2-49bb-b791-5b87e28d693b","Type":"ContainerStarted","Data":"e71754692a8d2e921e65f6b4b754ebbc7623bb12d83a060b0c884434190f54b7"} Oct 30 01:10:37.805605 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:37.805544 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9" Oct 30 01:10:37.848238 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:37.848184 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2" podStartSLOduration=1.848169314 podStartE2EDuration="1.848169314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:37.847064435 +0000 UTC m=+1638.162932761" watchObservedRunningTime="2026-10-30 01:10:37.848169314 +0000 UTC m=+1638.164037639" Oct 30 01:10:37.891262 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:37.891206 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9" podStartSLOduration=2.891189772 podStartE2EDuration="2.891189772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:37.88779107 +0000 UTC m=+1638.203659421" watchObservedRunningTime="2026-10-30 01:10:37.891189772 +0000 UTC m=+1638.207058097" Oct 30 01:10:38.267634 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:38.267594 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:39.759215 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:39.759176 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4"] Oct 30 01:10:39.763945 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:39.763915 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4" Oct 30 01:10:39.783127 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:39.783074 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4"] Oct 30 01:10:39.792229 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:39.792194 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-64f472-serving-cert\"" Oct 30 01:10:39.808502 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:39.808461 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4\" (UID: \"26ccc4a1-e4a8-492b-abef-87dd46d9dab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4" Oct 30 01:10:39.808870 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:39.808838 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4\" (UID: \"26ccc4a1-e4a8-492b-abef-87dd46d9dab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4" Oct 30 01:10:39.809237 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:39.809197 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4\" (UID: \"26ccc4a1-e4a8-492b-abef-87dd46d9dab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4" Oct 30 01:10:39.809340 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:39.809280 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzfww\" (UniqueName: \"kubernetes.io/projected/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-kube-api-access-rzfww\") pod \"maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4\" (UID: \"26ccc4a1-e4a8-492b-abef-87dd46d9dab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4" Oct 30 01:10:39.837704 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:39.837663 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:39.910930 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:39.910887 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4\" (UID: \"26ccc4a1-e4a8-492b-abef-87dd46d9dab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4" Oct 30 01:10:39.911146 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:39.910966 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzfww\" (UniqueName: \"kubernetes.io/projected/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-kube-api-access-rzfww\") pod \"maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4\" (UID: \"26ccc4a1-e4a8-492b-abef-87dd46d9dab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4" Oct 30 01:10:39.911146 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:39.911111 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4\" (UID: \"26ccc4a1-e4a8-492b-abef-87dd46d9dab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4" Oct 30 01:10:39.911271 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:39.911189 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4\" (UID: \"26ccc4a1-e4a8-492b-abef-87dd46d9dab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4" Oct 30 01:10:39.911879 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:39.911679 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4\" (UID: \"26ccc4a1-e4a8-492b-abef-87dd46d9dab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4" Oct 30 01:10:39.913695 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:39.913673 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4\" (UID: \"26ccc4a1-e4a8-492b-abef-87dd46d9dab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4" Oct 30 01:10:39.914095 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:39.914068 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4\" (UID: \"26ccc4a1-e4a8-492b-abef-87dd46d9dab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4" Oct 30 01:10:39.933501 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:39.933463 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:39.935015 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:39.934987 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzfww\" (UniqueName: \"kubernetes.io/projected/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-kube-api-access-rzfww\") pod \"maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4\" (UID: \"26ccc4a1-e4a8-492b-abef-87dd46d9dab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4" Oct 30 01:10:40.035499 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:40.035405 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:40.080653 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:40.080618 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4" Oct 30 01:10:40.397897 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:40.397862 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4"] Oct 30 01:10:40.398644 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:10:40.398616 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ccc4a1_e4a8_492b_abef_87dd46d9dab3.slice/crio-14287288fdecad25a6c2b3df47f4007a7ec0a70d002bf7c2278b2a629b606fbf WatchSource:0}: Error finding container 14287288fdecad25a6c2b3df47f4007a7ec0a70d002bf7c2278b2a629b606fbf: Status 404 returned error can't find the container with id 14287288fdecad25a6c2b3df47f4007a7ec0a70d002bf7c2278b2a629b606fbf Oct 30 01:10:40.513952 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:40.513915 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc"] Oct 30 01:10:40.623136 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:40.623101 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:40.722285 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:40.722253 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:40.850250 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:40.850012 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4" event={"ID":"26ccc4a1-e4a8-492b-abef-87dd46d9dab3","Type":"ContainerStarted","Data":"9a92b44b3beee98180f9f5a0651402d10b1933d1c054881ad9a91acd1edd8155"} Oct 30 01:10:40.850250 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:40.850078 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4" event={"ID":"26ccc4a1-e4a8-492b-abef-87dd46d9dab3","Type":"ContainerStarted","Data":"14287288fdecad25a6c2b3df47f4007a7ec0a70d002bf7c2278b2a629b606fbf"} Oct 30 01:10:40.851425 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:40.850302 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4" Oct 30 01:10:40.897349 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:40.897276 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4" podStartSLOduration=1.556090256 podStartE2EDuration="1.897254584s" totalImagesPullingTime="341.164328ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:40.894693343 +0000 UTC m=+1641.210561672" watchObservedRunningTime="2026-10-30 01:10:40.897254584 +0000 UTC m=+1641.213122910" Oct 30 01:10:41.656331 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:41.656298 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:43.614449 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:43.614396 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:45.276812 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:45.276739 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:45.514671 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:45.514623 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc" podUID="dee7a3e4-4f8c-41d0-b445-6c40118a8ae0" containerName="payload-pre-processing" containerID="cri-o://8909528784b1df69d9f0c3e5f4d026c8ed5c59d099333ffd4e1a1b859696705b" gracePeriod=25 Oct 30 01:10:45.808844 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:45.808813 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc" Oct 30 01:10:45.871877 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:45.871840 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dee7a3e4-4f8c-41d0-b445-6c40118a8ae0-kube-api-access-4j7mn\" (UniqueName: \"kubernetes.io/projected/dee7a3e4-4f8c-41d0-b445-6c40118a8ae0-kube-api-access-4j7mn\") pod \"dee7a3e4-4f8c-41d0-b445-6c40118a8ae0\" (UID: \"dee7a3e4-4f8c-41d0-b445-6c40118a8ae0\") " Oct 30 01:10:45.872255 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:45.872232 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dee7a3e4-4f8c-41d0-b445-6c40118a8ae0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dee7a3e4-4f8c-41d0-b445-6c40118a8ae0-plugins-config-volume\") pod \"dee7a3e4-4f8c-41d0-b445-6c40118a8ae0\" (UID: \"dee7a3e4-4f8c-41d0-b445-6c40118a8ae0\") " Oct 30 01:10:45.872675 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:45.872648 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dee7a3e4-4f8c-41d0-b445-6c40118a8ae0-plugins-config-volume" pod "dee7a3e4-4f8c-41d0-b445-6c40118a8ae0" (UID: "dee7a3e4-4f8c-41d0-b445-6c40118a8ae0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:10:45.872991 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:45.872969 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dee7a3e4-4f8c-41d0-b445-6c40118a8ae0-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:10:45.874519 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:45.874495 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dee7a3e4-4f8c-41d0-b445-6c40118a8ae0-kube-api-access-4j7mn" pod "dee7a3e4-4f8c-41d0-b445-6c40118a8ae0" (UID: "dee7a3e4-4f8c-41d0-b445-6c40118a8ae0"). InnerVolumeSpecName "kube-api-access-4j7mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:10:45.888096 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:45.888055 5300 generic.go:356] "Generic (PLEG): container finished" podID="dee7a3e4-4f8c-41d0-b445-6c40118a8ae0" containerID="8909528784b1df69d9f0c3e5f4d026c8ed5c59d099333ffd4e1a1b859696705b" exitCode=0 Oct 30 01:10:45.888288 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:45.888103 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc" Oct 30 01:10:45.888288 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:45.888191 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc" event={"ID":"dee7a3e4-4f8c-41d0-b445-6c40118a8ae0","Type":"ContainerDied","Data":"8909528784b1df69d9f0c3e5f4d026c8ed5c59d099333ffd4e1a1b859696705b"} Oct 30 01:10:45.888288 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:45.888239 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc" event={"ID":"dee7a3e4-4f8c-41d0-b445-6c40118a8ae0","Type":"ContainerDied","Data":"3e733a5e5c8f7a71d9aa39922ec35b80f85e97a804fc16077d1fa8a7ccbc4b95"} Oct 30 01:10:45.888288 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:45.888269 5300 scope.go:122] "RemoveContainer" containerID="8909528784b1df69d9f0c3e5f4d026c8ed5c59d099333ffd4e1a1b859696705b" Oct 30 01:10:45.908218 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:45.908194 5300 scope.go:122] "RemoveContainer" containerID="8909528784b1df69d9f0c3e5f4d026c8ed5c59d099333ffd4e1a1b859696705b" Oct 30 01:10:45.908606 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:10:45.908582 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8909528784b1df69d9f0c3e5f4d026c8ed5c59d099333ffd4e1a1b859696705b\": container with ID starting with 8909528784b1df69d9f0c3e5f4d026c8ed5c59d099333ffd4e1a1b859696705b not found: ID does not exist" containerID="8909528784b1df69d9f0c3e5f4d026c8ed5c59d099333ffd4e1a1b859696705b" Oct 30 01:10:45.908683 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:45.908620 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8909528784b1df69d9f0c3e5f4d026c8ed5c59d099333ffd4e1a1b859696705b"} err="failed to get container status \"8909528784b1df69d9f0c3e5f4d026c8ed5c59d099333ffd4e1a1b859696705b\": rpc error: code = NotFound desc = could not find container \"8909528784b1df69d9f0c3e5f4d026c8ed5c59d099333ffd4e1a1b859696705b\": container with ID starting with 8909528784b1df69d9f0c3e5f4d026c8ed5c59d099333ffd4e1a1b859696705b not found: ID does not exist" Oct 30 01:10:45.947208 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:45.947171 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc"] Oct 30 01:10:45.956245 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:45.956179 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da60de-dd5598cb7-fxnsc"] Oct 30 01:10:45.974032 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:45.974000 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j7mn\" (UniqueName: \"kubernetes.io/projected/dee7a3e4-4f8c-41d0-b445-6c40118a8ae0-kube-api-access-4j7mn\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:10:46.863509 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:46.863479 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4" Oct 30 01:10:46.947976 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:46.947939 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:47.822704 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:47.822666 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dee7a3e4-4f8c-41d0-b445-6c40118a8ae0" path="/var/lib/kubelet/pods/dee7a3e4-4f8c-41d0-b445-6c40118a8ae0/volumes" Oct 30 01:10:48.729388 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:48.729337 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:48.817167 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:48.817136 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2" Oct 30 01:10:48.817345 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:48.817252 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9" Oct 30 01:10:50.383717 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:50.383681 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:51.595170 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:10:51.595134 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:10:52.061382 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:52.061328 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:53.824980 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:53.824943 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:55.711280 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:55.711241 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:57.378079 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:57.378035 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:10:58.949802 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:10:58.949741 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:00.645185 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:00.645149 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:01.489361 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:01.489324 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:01.885165 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:01.885084 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:02.437380 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:02.437344 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:02.542449 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:02.542420 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:02.711838 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:02.705310 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:03.616045 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:03.616006 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:04.099504 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.099463 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:04.272112 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.272080 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr"] Oct 30 01:11:04.272987 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.272939 5300 cpu_manager.go:414] "containerMap: removing container" podUID="dee7a3e4-4f8c-41d0-b445-6c40118a8ae0" containerName="payload-pre-processing" Oct 30 01:11:04.272987 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.272961 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dee7a3e4-4f8c-41d0-b445-6c40118a8ae0" containerName="payload-pre-processing" Oct 30 01:11:04.273194 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.273176 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="dee7a3e4-4f8c-41d0-b445-6c40118a8ae0" containerName="payload-pre-processing" Oct 30 01:11:04.280008 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.279978 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.304511 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.304477 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr"] Oct 30 01:11:04.312031 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.312003 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-82e9bf-openshift-default-dockercfg-lnnf7\"" Oct 30 01:11:04.334166 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.334116 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:04.379805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.379667 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49cb6b12-e304-4889-b18e-2d9406bde9ac-istiod-ca-cert\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.379805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.379755 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-workload-socket\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.380086 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.380018 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-envoy\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.380154 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.380107 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrvvk\" (UniqueName: \"kubernetes.io/projected/49cb6b12-e304-4889-b18e-2d9406bde9ac-kube-api-access-vrvvk\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.380212 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.380178 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-podinfo\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.381084 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.381051 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-data\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.381217 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.381095 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-token\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.381217 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.381119 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-workload-certs\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.381217 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.381179 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-credential-socket\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.490782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.482909 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrvvk\" (UniqueName: \"kubernetes.io/projected/49cb6b12-e304-4889-b18e-2d9406bde9ac-kube-api-access-vrvvk\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.490782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.482964 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-podinfo\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.490782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.483058 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-data\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.490782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.483081 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-token\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.490782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.483104 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-workload-certs\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.490782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.483154 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-credential-socket\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.490782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.483217 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49cb6b12-e304-4889-b18e-2d9406bde9ac-istiod-ca-cert\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.490782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.483275 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-workload-socket\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.490782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.483411 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-envoy\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.490782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.486168 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-workload-certs\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.490782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.486750 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49cb6b12-e304-4889-b18e-2d9406bde9ac-istiod-ca-cert\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.490782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.487092 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-data\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.490782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.487393 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-workload-socket\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.490782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.487646 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-credential-socket\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.496312 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.492732 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-podinfo\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.515011 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.498577 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-envoy\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.515011 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.511117 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrvvk\" (UniqueName: \"kubernetes.io/projected/49cb6b12-e304-4889-b18e-2d9406bde9ac-kube-api-access-vrvvk\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.515011 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.511645 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-token\") pod \"e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.604465 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.604422 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:04.881876 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.881801 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:04.947612 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.947577 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr"] Oct 30 01:11:04.949891 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.949851 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:11:04.950018 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.949926 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:11:04.950018 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:04.949954 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:11:05.047824 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:05.047147 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" event={"ID":"49cb6b12-e304-4889-b18e-2d9406bde9ac","Type":"ContainerStarted","Data":"65c2b209448efc4379178649cd975370cdbabd35e2e329cdea945b60591e2edd"} Oct 30 01:11:05.047824 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:05.047209 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" event={"ID":"49cb6b12-e304-4889-b18e-2d9406bde9ac","Type":"ContainerStarted","Data":"a5c12d00bad3656df2e207a14deb7d1fe5cf540e1ffe35e85cd751dcb6187ff8"} Oct 30 01:11:05.094941 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:05.094882 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" podStartSLOduration=1.0948658 podStartE2EDuration="1.0948658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:05.093320244 +0000 UTC m=+1665.409188568" watchObservedRunningTime="2026-10-30 01:11:05.0948658 +0000 UTC m=+1665.410734124" Oct 30 01:11:05.605298 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:05.605264 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:06.066006 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:06.065977 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:06.611754 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:06.611720 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:06.647008 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:06.646967 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:06.751108 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:06.751074 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:06.851380 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:06.851348 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:07.068272 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:07.068226 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:07.069361 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:07.069339 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:11:07.443066 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:07.442979 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:07.634887 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:07.634847 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:07.740990 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:07.740952 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:07.846512 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:07.846478 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:08.100127 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:08.100032 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs"] Oct 30 01:11:08.105127 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:08.105099 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs" Oct 30 01:11:08.139387 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:08.139352 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs"] Oct 30 01:11:08.143549 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:08.143510 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-4dde48-serving-cert\"" Oct 30 01:11:08.209714 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:08.209675 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:08.250652 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:08.250599 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs\" (UID: \"2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs" Oct 30 01:11:08.250864 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:08.250731 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-maas-api-tls\") pod \"maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs\" (UID: \"2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs" Oct 30 01:11:08.251011 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:08.250983 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7mcd\" (UniqueName: \"kubernetes.io/projected/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-kube-api-access-v7mcd\") pod \"maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs\" (UID: \"2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs" Oct 30 01:11:08.251176 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:08.251160 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-metrics-certs\") pod \"maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs\" (UID: \"2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs" Oct 30 01:11:08.323336 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:08.323300 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:08.352805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:08.352676 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs\" (UID: \"2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs" Oct 30 01:11:08.352805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:08.352747 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-maas-api-tls\") pod \"maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs\" (UID: \"2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs" Oct 30 01:11:08.353048 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:08.352923 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7mcd\" (UniqueName: \"kubernetes.io/projected/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-kube-api-access-v7mcd\") pod \"maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs\" (UID: \"2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs" Oct 30 01:11:08.353124 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:08.353067 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-metrics-certs\") pod \"maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs\" (UID: \"2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs" Oct 30 01:11:08.353548 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:08.353521 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs\" (UID: \"2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs" Oct 30 01:11:08.355940 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:08.355914 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-metrics-certs\") pod \"maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs\" (UID: \"2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs" Oct 30 01:11:08.356126 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:08.356093 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-maas-api-tls\") pod \"maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs\" (UID: \"2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs" Oct 30 01:11:08.379855 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:08.379816 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7mcd\" (UniqueName: \"kubernetes.io/projected/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-kube-api-access-v7mcd\") pod \"maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs\" (UID: \"2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs" Oct 30 01:11:08.451277 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:08.451169 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:08.455801 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:08.455751 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs" Oct 30 01:11:08.556815 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:08.556750 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:08.940882 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:08.940730 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs"] Oct 30 01:11:08.950973 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:11:08.950935 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd50e96_1b82_4e81_9c21_dc35bd0e3ff9.slice/crio-dac2dc82a55958258371d6596dcc6375babf2c27c6a5a96d1d21c33ce5dde409 WatchSource:0}: Error finding container dac2dc82a55958258371d6596dcc6375babf2c27c6a5a96d1d21c33ce5dde409: Status 404 returned error can't find the container with id dac2dc82a55958258371d6596dcc6375babf2c27c6a5a96d1d21c33ce5dde409 Oct 30 01:11:09.083573 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:09.083532 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs" event={"ID":"2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9","Type":"ContainerStarted","Data":"dac2dc82a55958258371d6596dcc6375babf2c27c6a5a96d1d21c33ce5dde409"} Oct 30 01:11:09.456430 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:09.456391 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:09.561114 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:09.561023 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:09.651699 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:09.651661 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w"] Oct 30 01:11:09.656753 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:09.656726 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w" Oct 30 01:11:09.676017 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:09.675986 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w"] Oct 30 01:11:09.689387 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:09.688550 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-687471-dockercfg-26g74\"" Oct 30 01:11:09.689663 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:09.689642 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-687471\"" Oct 30 01:11:09.797786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:09.796885 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc41d7ad-a029-4258-9927-b29a5c8e6948-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w\" (UID: \"cc41d7ad-a029-4258-9927-b29a5c8e6948\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w" Oct 30 01:11:09.797786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:09.797233 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5mnv\" (UniqueName: \"kubernetes.io/projected/cc41d7ad-a029-4258-9927-b29a5c8e6948-kube-api-access-b5mnv\") pod \"payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w\" (UID: \"cc41d7ad-a029-4258-9927-b29a5c8e6948\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w" Oct 30 01:11:09.898553 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:09.898451 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5mnv\" (UniqueName: \"kubernetes.io/projected/cc41d7ad-a029-4258-9927-b29a5c8e6948-kube-api-access-b5mnv\") pod \"payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w\" (UID: \"cc41d7ad-a029-4258-9927-b29a5c8e6948\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w" Oct 30 01:11:09.898738 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:09.898609 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc41d7ad-a029-4258-9927-b29a5c8e6948-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w\" (UID: \"cc41d7ad-a029-4258-9927-b29a5c8e6948\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w" Oct 30 01:11:09.899471 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:09.899445 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc41d7ad-a029-4258-9927-b29a5c8e6948-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w\" (UID: \"cc41d7ad-a029-4258-9927-b29a5c8e6948\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w" Oct 30 01:11:09.919442 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:09.919410 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5mnv\" (UniqueName: \"kubernetes.io/projected/cc41d7ad-a029-4258-9927-b29a5c8e6948-kube-api-access-b5mnv\") pod \"payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w\" (UID: \"cc41d7ad-a029-4258-9927-b29a5c8e6948\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w" Oct 30 01:11:09.975538 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:09.975500 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w" Oct 30 01:11:10.112913 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:10.111595 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs" event={"ID":"2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9","Type":"ContainerStarted","Data":"1c408594fa159605ba6043627729d80c0f1ba5aea5d22c51257b7f6b31ccda5d"} Oct 30 01:11:10.112913 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:10.112846 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs" Oct 30 01:11:10.163874 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:10.163730 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs" podStartSLOduration=1.8052344740000001 podStartE2EDuration="2.16371286s" totalImagesPullingTime="358.478386ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:10.160881835 +0000 UTC m=+1670.476750155" watchObservedRunningTime="2026-10-30 01:11:10.16371286 +0000 UTC m=+1670.479581184" Oct 30 01:11:10.292970 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:10.292934 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w"] Oct 30 01:11:10.293137 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:11:10.293080 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc41d7ad_a029_4258_9927_b29a5c8e6948.slice/crio-2fd69213b9d5b4c3f27f6fb2ee924324c191fff4f34ffeea7229c342fe2cddd0 WatchSource:0}: Error finding container 2fd69213b9d5b4c3f27f6fb2ee924324c191fff4f34ffeea7229c342fe2cddd0: Status 404 returned error can't find the container with id 2fd69213b9d5b4c3f27f6fb2ee924324c191fff4f34ffeea7229c342fe2cddd0 Oct 30 01:11:10.577843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:10.577750 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:11.121167 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:11.121133 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w" event={"ID":"cc41d7ad-a029-4258-9927-b29a5c8e6948","Type":"ContainerStarted","Data":"4c99ed0c88bf2b4a7e2ce8016238dc8720487307833f6b5ebe678392ac1e87c5"} Oct 30 01:11:11.121377 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:11.121192 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w" event={"ID":"cc41d7ad-a029-4258-9927-b29a5c8e6948","Type":"ContainerStarted","Data":"2fd69213b9d5b4c3f27f6fb2ee924324c191fff4f34ffeea7229c342fe2cddd0"} Oct 30 01:11:11.163909 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:11.163853 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w" podStartSLOduration=2.163837354 podStartE2EDuration="2.163837354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:11.162601707 +0000 UTC m=+1671.478470032" watchObservedRunningTime="2026-10-30 01:11:11.163837354 +0000 UTC m=+1671.479705680" Oct 30 01:11:12.128879 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:12.128836 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w" Oct 30 01:11:12.293044 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:12.293009 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:12.390954 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:12.390834 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:12.431349 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:12.431308 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p"] Oct 30 01:11:12.436069 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:12.436039 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p" Oct 30 01:11:12.468441 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:12.468410 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p"] Oct 30 01:11:12.477707 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:12.477667 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-82e9bf\"" Oct 30 01:11:12.549948 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:12.549905 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a678742-88c9-4eba-a738-3a364b1be24e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p\" (UID: \"0a678742-88c9-4eba-a738-3a364b1be24e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p" Oct 30 01:11:12.550707 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:12.550056 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72dj7\" (UniqueName: \"kubernetes.io/projected/0a678742-88c9-4eba-a738-3a364b1be24e-kube-api-access-72dj7\") pod \"payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p\" (UID: \"0a678742-88c9-4eba-a738-3a364b1be24e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p" Oct 30 01:11:12.691021 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:12.690917 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a678742-88c9-4eba-a738-3a364b1be24e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p\" (UID: \"0a678742-88c9-4eba-a738-3a364b1be24e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p" Oct 30 01:11:12.691021 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:12.691010 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72dj7\" (UniqueName: \"kubernetes.io/projected/0a678742-88c9-4eba-a738-3a364b1be24e-kube-api-access-72dj7\") pod \"payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p\" (UID: \"0a678742-88c9-4eba-a738-3a364b1be24e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p" Oct 30 01:11:12.692440 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:12.692410 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a678742-88c9-4eba-a738-3a364b1be24e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p\" (UID: \"0a678742-88c9-4eba-a738-3a364b1be24e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p" Oct 30 01:11:12.714782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:12.709841 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:12.714782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:12.713829 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72dj7\" (UniqueName: \"kubernetes.io/projected/0a678742-88c9-4eba-a738-3a364b1be24e-kube-api-access-72dj7\") pod \"payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p\" (UID: \"0a678742-88c9-4eba-a738-3a364b1be24e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p" Oct 30 01:11:12.767785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:12.757933 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p" Oct 30 01:11:12.825341 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:12.825308 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:12.947285 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:12.947207 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:13.106227 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:13.106195 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p"] Oct 30 01:11:13.106468 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:11:13.106446 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a678742_88c9_4eba_a738_3a364b1be24e.slice/crio-380fa9279d018bb63b532a0ea1204d6fa8f689125a3f8671f7e71f1109d56c5c WatchSource:0}: Error finding container 380fa9279d018bb63b532a0ea1204d6fa8f689125a3f8671f7e71f1109d56c5c: Status 404 returned error can't find the container with id 380fa9279d018bb63b532a0ea1204d6fa8f689125a3f8671f7e71f1109d56c5c Oct 30 01:11:13.139574 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:13.139545 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p" event={"ID":"0a678742-88c9-4eba-a738-3a364b1be24e","Type":"ContainerStarted","Data":"380fa9279d018bb63b532a0ea1204d6fa8f689125a3f8671f7e71f1109d56c5c"} Oct 30 01:11:14.150041 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:14.150007 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p" event={"ID":"0a678742-88c9-4eba-a738-3a364b1be24e","Type":"ContainerStarted","Data":"6da76c9118d7ae1ebde11505a74bdf00014f0b13f6d878570cedd89a06cf4b2f"} Oct 30 01:11:14.150561 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:14.150075 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p" Oct 30 01:11:14.195752 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:14.195693 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p" podStartSLOduration=2.195676282 podStartE2EDuration="2.195676282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:14.192845549 +0000 UTC m=+1674.508713875" watchObservedRunningTime="2026-10-30 01:11:14.195676282 +0000 UTC m=+1674.511544642" Oct 30 01:11:14.457612 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:14.457525 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:14.557882 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:14.557849 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:14.655370 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:14.655331 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9"] Oct 30 01:11:14.659687 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:14.659663 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9" Oct 30 01:11:14.675882 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:14.675840 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-56d1d9\"" Oct 30 01:11:14.678827 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:14.678805 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9"] Oct 30 01:11:14.793788 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:14.759182 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34e36fa-c21d-4f52-bc2e-5048d21414a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9\" (UID: \"b34e36fa-c21d-4f52-bc2e-5048d21414a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9" Oct 30 01:11:14.793788 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:14.759291 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc4s6\" (UniqueName: \"kubernetes.io/projected/b34e36fa-c21d-4f52-bc2e-5048d21414a7-kube-api-access-fc4s6\") pod \"payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9\" (UID: \"b34e36fa-c21d-4f52-bc2e-5048d21414a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9" Oct 30 01:11:14.793788 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:14.771156 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-56d1d9-dockercfg-f6lgf\"" Oct 30 01:11:14.898787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:14.888641 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc4s6\" (UniqueName: \"kubernetes.io/projected/b34e36fa-c21d-4f52-bc2e-5048d21414a7-kube-api-access-fc4s6\") pod \"payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9\" (UID: \"b34e36fa-c21d-4f52-bc2e-5048d21414a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9" Oct 30 01:11:14.898787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:14.889047 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34e36fa-c21d-4f52-bc2e-5048d21414a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9\" (UID: \"b34e36fa-c21d-4f52-bc2e-5048d21414a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9" Oct 30 01:11:14.908842 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:14.903555 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:14.908842 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:14.905973 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34e36fa-c21d-4f52-bc2e-5048d21414a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9\" (UID: \"b34e36fa-c21d-4f52-bc2e-5048d21414a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9" Oct 30 01:11:14.915712 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:14.915674 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc4s6\" (UniqueName: \"kubernetes.io/projected/b34e36fa-c21d-4f52-bc2e-5048d21414a7-kube-api-access-fc4s6\") pod \"payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9\" (UID: \"b34e36fa-c21d-4f52-bc2e-5048d21414a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9" Oct 30 01:11:14.974935 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:14.974902 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9" Oct 30 01:11:15.300696 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:15.300653 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9"] Oct 30 01:11:16.167228 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:16.167195 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9" event={"ID":"b34e36fa-c21d-4f52-bc2e-5048d21414a7","Type":"ContainerStarted","Data":"7a6d0681973ab48ca0eec0f429e3acedd08eec8cace9eaa4812ca0d0bfa26905"} Oct 30 01:11:16.167423 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:16.167252 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9" event={"ID":"b34e36fa-c21d-4f52-bc2e-5048d21414a7","Type":"ContainerStarted","Data":"93aa4c9aa0030276561fe7d7fd4cddf1097d31f49b8577e07846ea2d7fefd20a"} Oct 30 01:11:16.167423 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:16.167283 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9" Oct 30 01:11:16.210472 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:16.210408 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9" podStartSLOduration=2.210384119 podStartE2EDuration="2.210384119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:16.207611154 +0000 UTC m=+1676.523479481" watchObservedRunningTime="2026-10-30 01:11:16.210384119 +0000 UTC m=+1676.526252444" Oct 30 01:11:16.898258 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:16.898224 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:17.135069 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:17.135041 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs" Oct 30 01:11:18.755320 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:18.755276 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:19.672426 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:19.672382 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:19.999752 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:19.999709 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:20.454637 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:20.454530 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:20.538855 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:20.538823 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:20.727870 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:20.727828 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:20.751693 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:20.751542 5300 scope.go:122] "RemoveContainer" containerID="951e706344460305c279f5b991194c967e6163fdea439c2c86266afb73496c7c" Oct 30 01:11:21.204844 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:21.198266 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:21.441783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:21.441713 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:21.533520 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:21.533486 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:21.601670 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:11:21.601299 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:11:22.886927 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:22.886890 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:23.142862 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:23.142781 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w" Oct 30 01:11:24.748935 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:24.748894 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:24.865018 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:24.864985 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:24.966274 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:24.966226 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:25.161922 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:25.161828 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p" Oct 30 01:11:26.496059 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:26.496018 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:26.595276 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:26.595240 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:26.919273 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:26.919182 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7"] Oct 30 01:11:26.926944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:26.926908 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:26.967561 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:26.967530 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7" Oct 30 01:11:26.988268 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:26.988239 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-29145c\"" Oct 30 01:11:26.991346 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:26.991317 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7"] Oct 30 01:11:27.008786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:27.006536 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-29145c-dockercfg-5r9cj\"" Oct 30 01:11:27.101420 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:27.101376 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83ca74f3-ea7d-46b6-b954-f681f39ba286-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7\" (UID: \"83ca74f3-ea7d-46b6-b954-f681f39ba286\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7" Oct 30 01:11:27.101709 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:27.101680 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gr68\" (UniqueName: \"kubernetes.io/projected/83ca74f3-ea7d-46b6-b954-f681f39ba286-kube-api-access-4gr68\") pod \"payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7\" (UID: \"83ca74f3-ea7d-46b6-b954-f681f39ba286\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7" Oct 30 01:11:27.180300 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:27.180212 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9" Oct 30 01:11:27.202561 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:27.202531 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gr68\" (UniqueName: \"kubernetes.io/projected/83ca74f3-ea7d-46b6-b954-f681f39ba286-kube-api-access-4gr68\") pod \"payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7\" (UID: \"83ca74f3-ea7d-46b6-b954-f681f39ba286\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7" Oct 30 01:11:27.202921 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:27.202885 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83ca74f3-ea7d-46b6-b954-f681f39ba286-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7\" (UID: \"83ca74f3-ea7d-46b6-b954-f681f39ba286\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7" Oct 30 01:11:27.203716 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:27.203691 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83ca74f3-ea7d-46b6-b954-f681f39ba286-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7\" (UID: \"83ca74f3-ea7d-46b6-b954-f681f39ba286\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7" Oct 30 01:11:27.226641 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:27.226615 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gr68\" (UniqueName: \"kubernetes.io/projected/83ca74f3-ea7d-46b6-b954-f681f39ba286-kube-api-access-4gr68\") pod \"payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7\" (UID: \"83ca74f3-ea7d-46b6-b954-f681f39ba286\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7" Oct 30 01:11:27.297858 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:27.297829 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7" Oct 30 01:11:27.620652 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:27.620622 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7"] Oct 30 01:11:27.621059 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:11:27.620918 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ca74f3_ea7d_46b6_b954_f681f39ba286.slice/crio-07e58a422e6fd6df3f5ea8f51156e3a99747c35dfc33788d2d055aec357412a2 WatchSource:0}: Error finding container 07e58a422e6fd6df3f5ea8f51156e3a99747c35dfc33788d2d055aec357412a2: Status 404 returned error can't find the container with id 07e58a422e6fd6df3f5ea8f51156e3a99747c35dfc33788d2d055aec357412a2 Oct 30 01:11:28.399507 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:28.399468 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7" event={"ID":"83ca74f3-ea7d-46b6-b954-f681f39ba286","Type":"ContainerStarted","Data":"89f9f8e5b2a4b26e049efc989b8682bf971ac3b6d804e576f57651de3a636a97"} Oct 30 01:11:28.399707 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:28.399530 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7" event={"ID":"83ca74f3-ea7d-46b6-b954-f681f39ba286","Type":"ContainerStarted","Data":"07e58a422e6fd6df3f5ea8f51156e3a99747c35dfc33788d2d055aec357412a2"} Oct 30 01:11:28.399707 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:28.399558 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7" Oct 30 01:11:28.464281 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:28.464223 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7" podStartSLOduration=2.464206897 podStartE2EDuration="2.464206897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:28.460684493 +0000 UTC m=+1688.776552818" watchObservedRunningTime="2026-10-30 01:11:28.464206897 +0000 UTC m=+1688.780075221" Oct 30 01:11:28.676392 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:28.676306 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:30.365298 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:30.365262 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:32.034223 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:32.034188 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:32.846554 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:32.846510 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:33.738497 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:33.738459 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:35.453970 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:35.453925 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:36.914294 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:36.914257 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:37.211884 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:37.211740 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:39.145428 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:39.145391 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:39.411500 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:39.411414 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7" Oct 30 01:11:40.939885 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:40.939833 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:42.652957 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:42.652930 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:44.218358 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:44.218314 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:44.302655 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:44.302623 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:44.418066 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:44.418021 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:45.682720 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:45.682681 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:46.178297 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:46.178263 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:46.411784 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:46.387122 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:46.500507 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:46.500475 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:46.789526 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:46.789431 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:47.677242 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:47.677184 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:47.781503 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:47.781468 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:48.509819 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:48.508502 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:48.587154 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:48.587120 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:48.984611 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:48.984576 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:50.455092 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:50.455038 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:51.596521 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:11:51.596483 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:11:52.339881 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:52.339839 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:53.395403 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:53.395359 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:53.502189 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:53.502134 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:53.741308 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:53.741273 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:54.394954 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:54.394916 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:54.478653 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:54.478617 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:54.582130 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:54.582089 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:54.913134 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:54.913095 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j"] Oct 30 01:11:54.919317 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:54.919288 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:54.941282 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:54.941251 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j"] Oct 30 01:11:54.949312 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:54.949278 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-50d078-openshift-default-dockercfg-v9gcb\"" Oct 30 01:11:54.959597 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:54.959531 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:55.097801 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.090878 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgk57\" (UniqueName: \"kubernetes.io/projected/19cf7721-345d-4937-b699-292dcdde5ef0-kube-api-access-vgk57\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.097801 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.090952 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19cf7721-345d-4937-b699-292dcdde5ef0-istio-podinfo\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.097801 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.091096 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-istio-envoy\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.097801 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.091274 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19cf7721-345d-4937-b699-292dcdde5ef0-istio-token\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.097801 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.091344 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-workload-certs\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.097801 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.091483 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19cf7721-345d-4937-b699-292dcdde5ef0-istiod-ca-cert\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.097801 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.091561 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-workload-socket\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.097801 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.091588 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-istio-data\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.097801 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.091626 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-credential-socket\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.124628 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.124419 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:55.198488 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.198393 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-workload-socket\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.198488 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.198459 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-istio-data\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.198728 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.198499 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-credential-socket\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.198728 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.198568 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgk57\" (UniqueName: \"kubernetes.io/projected/19cf7721-345d-4937-b699-292dcdde5ef0-kube-api-access-vgk57\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.198728 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.198624 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19cf7721-345d-4937-b699-292dcdde5ef0-istio-podinfo\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.198925 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.198772 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-istio-envoy\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.198985 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.198952 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19cf7721-345d-4937-b699-292dcdde5ef0-istio-token\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.199059 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.199038 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-workload-certs\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.199244 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.199226 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19cf7721-345d-4937-b699-292dcdde5ef0-istiod-ca-cert\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.210819 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.200386 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19cf7721-345d-4937-b699-292dcdde5ef0-istiod-ca-cert\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.210819 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.200616 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-workload-socket\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.210819 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.200818 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-istio-data\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.210819 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.200999 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-credential-socket\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.210819 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.204237 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-workload-certs\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.210819 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.205158 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19cf7721-345d-4937-b699-292dcdde5ef0-istio-podinfo\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.232143 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.225093 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19cf7721-345d-4937-b699-292dcdde5ef0-istio-token\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.252784 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.245177 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-istio-envoy\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.255695 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.255598 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgk57\" (UniqueName: \"kubernetes.io/projected/19cf7721-345d-4937-b699-292dcdde5ef0-kube-api-access-vgk57\") pod \"e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.487197 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.487160 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:55.540953 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.540919 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:55.880304 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.880278 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j"] Oct 30 01:11:55.881218 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:11:55.881189 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19cf7721_345d_4937_b699_292dcdde5ef0.slice/crio-5b106501aa15d1467af516019b9686021254f47dbbdd028d72d54c470bb08514 WatchSource:0}: Error finding container 5b106501aa15d1467af516019b9686021254f47dbbdd028d72d54c470bb08514: Status 404 returned error can't find the container with id 5b106501aa15d1467af516019b9686021254f47dbbdd028d72d54c470bb08514 Oct 30 01:11:55.884143 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.884109 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:11:55.884231 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.884176 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:11:55.884231 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.884208 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:11:55.912437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:55.912411 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:56.280182 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:56.280147 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:56.386858 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:56.386818 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:56.538849 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:56.537432 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:56.646407 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:56.646369 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" event={"ID":"19cf7721-345d-4937-b699-292dcdde5ef0","Type":"ContainerStarted","Data":"e80351d009fc147f9cf2d48c73e241c831642b61befc5cd9629fd51da54e82bc"} Oct 30 01:11:56.646635 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:56.646426 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" event={"ID":"19cf7721-345d-4937-b699-292dcdde5ef0","Type":"ContainerStarted","Data":"5b106501aa15d1467af516019b9686021254f47dbbdd028d72d54c470bb08514"} Oct 30 01:11:56.700959 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:56.700883 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" podStartSLOduration=2.700860074 podStartE2EDuration="2.700860074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:56.696435843 +0000 UTC m=+1717.012304169" watchObservedRunningTime="2026-10-30 01:11:56.700860074 +0000 UTC m=+1717.016728401" Oct 30 01:11:57.542827 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:57.542148 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:57.602784 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:57.595608 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:57.687781 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:57.684412 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:57.687781 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:57.685672 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:11:58.078245 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:58.078207 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:59.260192 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:59.260156 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:11:59.368630 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:11:59.368592 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:00.135138 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:00.135094 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:00.990218 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:00.990133 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:01.087694 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:01.087657 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:01.332745 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:01.332657 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:01.479908 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:01.461955 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:02.190230 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:02.190191 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:03.188164 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:03.188124 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:03.284313 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:03.284282 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:04.041700 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:04.041660 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:05.785107 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:05.785070 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:07.437190 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:07.437161 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:07.533563 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:07.533533 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:07.733996 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:07.733945 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:08.203912 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:08.203829 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd"] Oct 30 01:12:08.204066 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:08.204046 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" podUID="d1f758e7-acc5-4a72-84cc-c70e23841347" containerName="istio-proxy" containerID="cri-o://6010d7ae1e66318d7251f66cab044d4c267ebdbfc049347df134d4565c727f89" gracePeriod=30 Oct 30 01:12:09.446646 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:09.446591 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:10.876118 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:10.876070 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:11.421512 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:11.421468 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:11.519506 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:11.519468 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:11.618652 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:11.618608 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:12.002610 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:12.002575 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:13.369308 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.369264 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:13.528324 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.528200 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 01:12:13.603731 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.603694 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-workload-certs\") pod \"d1f758e7-acc5-4a72-84cc-c70e23841347\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " Oct 30 01:12:13.603975 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.603833 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-podinfo\") pod \"d1f758e7-acc5-4a72-84cc-c70e23841347\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " Oct 30 01:12:13.603975 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.603915 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1f758e7-acc5-4a72-84cc-c70e23841347-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1f758e7-acc5-4a72-84cc-c70e23841347-istiod-ca-cert\") pod \"d1f758e7-acc5-4a72-84cc-c70e23841347\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " Oct 30 01:12:13.604105 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.604030 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-envoy\") pod \"d1f758e7-acc5-4a72-84cc-c70e23841347\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " Oct 30 01:12:13.604167 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.604135 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-credential-socket\") pod \"d1f758e7-acc5-4a72-84cc-c70e23841347\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " Oct 30 01:12:13.604276 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.604256 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-workload-socket\") pod \"d1f758e7-acc5-4a72-84cc-c70e23841347\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " Oct 30 01:12:13.604382 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.604337 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-data\") pod \"d1f758e7-acc5-4a72-84cc-c70e23841347\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " Oct 30 01:12:13.604466 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.604416 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1f758e7-acc5-4a72-84cc-c70e23841347-kube-api-access-9699f\" (UniqueName: \"kubernetes.io/projected/d1f758e7-acc5-4a72-84cc-c70e23841347-kube-api-access-9699f\") pod \"d1f758e7-acc5-4a72-84cc-c70e23841347\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " Oct 30 01:12:13.604547 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.604534 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-token\" (UniqueName: \"kubernetes.io/projected/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-token\") pod \"d1f758e7-acc5-4a72-84cc-c70e23841347\" (UID: \"d1f758e7-acc5-4a72-84cc-c70e23841347\") " Oct 30 01:12:13.604820 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.604751 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1f758e7-acc5-4a72-84cc-c70e23841347-istiod-ca-cert" pod "d1f758e7-acc5-4a72-84cc-c70e23841347" (UID: "d1f758e7-acc5-4a72-84cc-c70e23841347"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:12:13.605112 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.605089 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-workload-certs" pod "d1f758e7-acc5-4a72-84cc-c70e23841347" (UID: "d1f758e7-acc5-4a72-84cc-c70e23841347"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:13.605301 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.605285 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:13.605364 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.605305 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1f758e7-acc5-4a72-84cc-c70e23841347-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:13.607185 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.607157 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-envoy" pod "d1f758e7-acc5-4a72-84cc-c70e23841347" (UID: "d1f758e7-acc5-4a72-84cc-c70e23841347"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:13.607403 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.607373 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-credential-socket" pod "d1f758e7-acc5-4a72-84cc-c70e23841347" (UID: "d1f758e7-acc5-4a72-84cc-c70e23841347"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:13.607525 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.607504 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-workload-socket" pod "d1f758e7-acc5-4a72-84cc-c70e23841347" (UID: "d1f758e7-acc5-4a72-84cc-c70e23841347"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:13.607847 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.607678 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-data" pod "d1f758e7-acc5-4a72-84cc-c70e23841347" (UID: "d1f758e7-acc5-4a72-84cc-c70e23841347"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:13.610701 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.610669 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f758e7-acc5-4a72-84cc-c70e23841347-kube-api-access-9699f" pod "d1f758e7-acc5-4a72-84cc-c70e23841347" (UID: "d1f758e7-acc5-4a72-84cc-c70e23841347"). InnerVolumeSpecName "kube-api-access-9699f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:13.615864 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.612728 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-podinfo" pod "d1f758e7-acc5-4a72-84cc-c70e23841347" (UID: "d1f758e7-acc5-4a72-84cc-c70e23841347"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:12:13.615864 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.613118 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-token" pod "d1f758e7-acc5-4a72-84cc-c70e23841347" (UID: "d1f758e7-acc5-4a72-84cc-c70e23841347"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:13.706698 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.706665 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:13.706698 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.706694 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:13.706698 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.706705 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:13.707041 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.706713 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:13.707041 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.706722 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9699f\" (UniqueName: \"kubernetes.io/projected/d1f758e7-acc5-4a72-84cc-c70e23841347-kube-api-access-9699f\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:13.707041 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.706731 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:13.707041 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.706739 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1f758e7-acc5-4a72-84cc-c70e23841347-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:13.818255 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.818145 5300 generic.go:356] "Generic (PLEG): container finished" podID="d1f758e7-acc5-4a72-84cc-c70e23841347" containerID="6010d7ae1e66318d7251f66cab044d4c267ebdbfc049347df134d4565c727f89" exitCode=0 Oct 30 01:12:13.818255 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.818246 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" Oct 30 01:12:13.833679 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.833635 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" event={"ID":"d1f758e7-acc5-4a72-84cc-c70e23841347","Type":"ContainerDied","Data":"6010d7ae1e66318d7251f66cab044d4c267ebdbfc049347df134d4565c727f89"} Oct 30 01:12:13.833819 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.833685 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd" event={"ID":"d1f758e7-acc5-4a72-84cc-c70e23841347","Type":"ContainerDied","Data":"0b218986550c01219dfab921a253031f96f8b52178d54ae5ae375d77e80be7d7"} Oct 30 01:12:13.833819 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.833709 5300 scope.go:122] "RemoveContainer" containerID="6010d7ae1e66318d7251f66cab044d4c267ebdbfc049347df134d4565c727f89" Oct 30 01:12:13.847613 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.847583 5300 scope.go:122] "RemoveContainer" containerID="6010d7ae1e66318d7251f66cab044d4c267ebdbfc049347df134d4565c727f89" Oct 30 01:12:13.847948 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:12:13.847927 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6010d7ae1e66318d7251f66cab044d4c267ebdbfc049347df134d4565c727f89\": container with ID starting with 6010d7ae1e66318d7251f66cab044d4c267ebdbfc049347df134d4565c727f89 not found: ID does not exist" containerID="6010d7ae1e66318d7251f66cab044d4c267ebdbfc049347df134d4565c727f89" Oct 30 01:12:13.848027 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.847957 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6010d7ae1e66318d7251f66cab044d4c267ebdbfc049347df134d4565c727f89"} err="failed to get container status \"6010d7ae1e66318d7251f66cab044d4c267ebdbfc049347df134d4565c727f89\": rpc error: code = NotFound desc = could not find container \"6010d7ae1e66318d7251f66cab044d4c267ebdbfc049347df134d4565c727f89\": container with ID starting with 6010d7ae1e66318d7251f66cab044d4c267ebdbfc049347df134d4565c727f89 not found: ID does not exist" Oct 30 01:12:13.882494 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.882458 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd"] Oct 30 01:12:13.886012 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:13.885982 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-vgpvd"] Oct 30 01:12:15.403125 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:15.403086 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:15.823073 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:15.823040 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1f758e7-acc5-4a72-84cc-c70e23841347" path="/var/lib/kubelet/pods/d1f758e7-acc5-4a72-84cc-c70e23841347/volumes" Oct 30 01:12:17.162265 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:17.162223 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:18.277689 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:18.277652 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:18.392292 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:18.392245 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:18.771065 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:18.771019 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:18.994201 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:18.994163 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:21.142798 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:21.142518 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:21.595249 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:12:21.595211 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:12:23.045660 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:23.045605 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:24.872223 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:24.872184 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:26.735124 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:26.735077 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:28.360298 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:28.360258 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:29.842169 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:29.842131 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:29.978219 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:29.978178 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:30.172456 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:30.172418 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:31.071433 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:31.071399 5300 generic.go:356] "Generic (PLEG): container finished" podID="49cb6b12-e304-4889-b18e-2d9406bde9ac" containerID="65c2b209448efc4379178649cd975370cdbabd35e2e329cdea945b60591e2edd" exitCode=0 Oct 30 01:12:31.071851 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:31.071476 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" event={"ID":"49cb6b12-e304-4889-b18e-2d9406bde9ac","Type":"ContainerDied","Data":"65c2b209448efc4379178649cd975370cdbabd35e2e329cdea945b60591e2edd"} Oct 30 01:12:31.071981 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:31.071966 5300 scope.go:122] "RemoveContainer" containerID="65c2b209448efc4379178649cd975370cdbabd35e2e329cdea945b60591e2edd" Oct 30 01:12:31.074051 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:31.074021 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:12:31.074137 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:31.074094 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:12:31.074137 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:31.074130 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:12:32.084119 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:32.084078 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" event={"ID":"49cb6b12-e304-4889-b18e-2d9406bde9ac","Type":"ContainerStarted","Data":"6e19b1869bf29fb07181a0cdb247413a4290d4ef5bfe5e0dfaedca803c59ddac"} Oct 30 01:12:32.165277 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:32.165242 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:32.605673 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:32.605627 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:12:32.606982 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:32.606954 5300 patch_prober.go:29] interesting pod/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 30 01:12:32.607128 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:32.607003 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" podUID="49cb6b12-e304-4889-b18e-2d9406bde9ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 30 01:12:32.896052 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:32.895963 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:33.494032 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:33.493997 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:33.605273 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:33.605236 5300 patch_prober.go:29] interesting pod/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 30 01:12:33.605480 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:33.605296 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" podUID="49cb6b12-e304-4889-b18e-2d9406bde9ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 30 01:12:33.871840 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:33.871598 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:34.064407 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:34.064371 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:34.480470 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:34.480427 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:34.607158 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:34.607121 5300 patch_prober.go:29] interesting pod/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 30 01:12:34.607677 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:34.607191 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" podUID="49cb6b12-e304-4889-b18e-2d9406bde9ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 30 01:12:34.939845 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:34.939325 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:35.051414 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:35.051372 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:35.607075 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:35.605750 5300 patch_prober.go:29] interesting pod/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 30 01:12:35.607075 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:35.605834 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" podUID="49cb6b12-e304-4889-b18e-2d9406bde9ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 30 01:12:35.914634 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:35.914548 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8"] Oct 30 01:12:35.915936 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:35.915909 5300 cpu_manager.go:414] "containerMap: removing container" podUID="d1f758e7-acc5-4a72-84cc-c70e23841347" containerName="istio-proxy" Oct 30 01:12:35.916086 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:35.916074 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1f758e7-acc5-4a72-84cc-c70e23841347" containerName="istio-proxy" Oct 30 01:12:35.916360 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:35.916347 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1f758e7-acc5-4a72-84cc-c70e23841347" containerName="istio-proxy" Oct 30 01:12:35.980362 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:35.979818 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:35.983114 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:35.983082 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:35.987458 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:35.987427 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8"] Oct 30 01:12:36.003319 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.003281 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e69dee-openshift-default-dockercfg-lxjwb\"" Oct 30 01:12:36.066790 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.065454 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-envoy\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.066790 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.065531 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-token\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.066790 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.065561 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c908ef75-1950-4c44-8311-47bcfd6b814a-istiod-ca-cert\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.066790 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.065723 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-credential-socket\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.066790 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.065755 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-workload-socket\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.066790 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.065843 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-workload-certs\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.066790 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.065972 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-podinfo\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.066790 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.066053 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-data\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.066790 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.066125 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxf4b\" (UniqueName: \"kubernetes.io/projected/c908ef75-1950-4c44-8311-47bcfd6b814a-kube-api-access-bxf4b\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.093024 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.092990 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:36.167301 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.167216 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-envoy\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.167301 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.167268 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-token\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.167301 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.167286 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c908ef75-1950-4c44-8311-47bcfd6b814a-istiod-ca-cert\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.167545 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.167343 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-credential-socket\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.167545 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.167361 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-workload-socket\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.167545 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.167396 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-workload-certs\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.167545 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.167466 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-podinfo\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.167545 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.167539 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-data\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.167726 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.167604 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxf4b\" (UniqueName: \"kubernetes.io/projected/c908ef75-1950-4c44-8311-47bcfd6b814a-kube-api-access-bxf4b\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.168374 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.168345 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-workload-socket\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.168486 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.168464 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-data\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.168701 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.168679 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-workload-certs\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.168776 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.168691 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-credential-socket\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.173964 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.173924 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c908ef75-1950-4c44-8311-47bcfd6b814a-istiod-ca-cert\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.179511 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.179475 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-envoy\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.180369 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.180344 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-podinfo\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.198406 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.198207 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxf4b\" (UniqueName: \"kubernetes.io/projected/c908ef75-1950-4c44-8311-47bcfd6b814a-kube-api-access-bxf4b\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.203271 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.203238 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-token\") pod \"e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.311931 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.311893 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:36.464635 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.464492 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:36.605044 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.605014 5300 patch_prober.go:29] interesting pod/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 30 01:12:36.605296 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.605246 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" podUID="49cb6b12-e304-4889-b18e-2d9406bde9ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 30 01:12:36.749720 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.749684 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8"] Oct 30 01:12:36.751969 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:12:36.751935 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc908ef75_1950_4c44_8311_47bcfd6b814a.slice/crio-fea5702ba754a68c528e3fc9cac1da99bfa6a20556c6540d6e7375113d2b76db WatchSource:0}: Error finding container fea5702ba754a68c528e3fc9cac1da99bfa6a20556c6540d6e7375113d2b76db: Status 404 returned error can't find the container with id fea5702ba754a68c528e3fc9cac1da99bfa6a20556c6540d6e7375113d2b76db Oct 30 01:12:36.755659 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.755616 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:12:36.755810 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.755697 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:12:36.755810 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:36.755731 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:12:37.068278 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:37.068159 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:12:37.168786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:37.164950 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" event={"ID":"c908ef75-1950-4c44-8311-47bcfd6b814a","Type":"ContainerStarted","Data":"100d055f2312e10c435768d93aefdefa6a92e5c3e5682a40b472c1cda81a0e1d"} Oct 30 01:12:37.168786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:37.165057 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" event={"ID":"c908ef75-1950-4c44-8311-47bcfd6b814a","Type":"ContainerStarted","Data":"fea5702ba754a68c528e3fc9cac1da99bfa6a20556c6540d6e7375113d2b76db"} Oct 30 01:12:37.245750 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:37.238927 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" podStartSLOduration=2.238903747 podStartE2EDuration="2.238903747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:37.213941235 +0000 UTC m=+1757.529809560" watchObservedRunningTime="2026-10-30 01:12:37.238903747 +0000 UTC m=+1757.554772072" Oct 30 01:12:37.313285 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:37.313249 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:37.315162 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:37.315133 5300 patch_prober.go:29] interesting pod/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 01:12:37.315305 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:37.315196 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" podUID="c908ef75-1950-4c44-8311-47bcfd6b814a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 01:12:37.605140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:37.605106 5300 patch_prober.go:29] interesting pod/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 30 01:12:37.605379 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:37.605160 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" podUID="49cb6b12-e304-4889-b18e-2d9406bde9ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 30 01:12:37.852206 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:37.852172 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm"] Oct 30 01:12:37.852700 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:37.852385 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" podUID="7bd004c0-3447-4382-82f5-e797f86b850e" containerName="istio-proxy" containerID="cri-o://017ed38f55ce4b5a817d8e5863c39dafb727cc66d33f24b33d6bb300359b470b" gracePeriod=30 Oct 30 01:12:38.033437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:38.033398 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn"] Oct 30 01:12:38.033692 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:38.033638 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" podUID="22aacbb7-595e-4895-bcd4-5fec9c9d78e9" containerName="istio-proxy" containerID="cri-o://a6ef06136594319e95049c1f8719e92eb131384c1ba6f735997606f88a6195c4" gracePeriod=30 Oct 30 01:12:38.320252 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:38.320167 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:38.373001 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:38.372959 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:38.458476 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:38.452427 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2"] Oct 30 01:12:38.462623 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:38.459913 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" podUID="59536065-cfa0-4f0c-a2ca-55284c369d81" containerName="istio-proxy" containerID="cri-o://42661b1ed5c2d29b94d756ac672ebbaf0990f6d22d30b33c09dbf42ddf99c06f" gracePeriod=30 Oct 30 01:12:38.614929 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:38.614823 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:12:38.616534 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:38.616505 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:12:39.183922 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:39.183888 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:39.184923 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:39.184899 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:12:39.718305 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:39.718259 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:39.916584 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:39.916540 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:40.030400 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:40.030361 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:40.520111 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:40.520071 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:42.334802 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:42.334713 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:42.439958 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:42.439926 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:42.535313 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:42.535270 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:43.252669 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.252286 5300 generic.go:356] "Generic (PLEG): container finished" podID="22aacbb7-595e-4895-bcd4-5fec9c9d78e9" containerID="a6ef06136594319e95049c1f8719e92eb131384c1ba6f735997606f88a6195c4" exitCode=0 Oct 30 01:12:43.252669 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.252593 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" event={"ID":"22aacbb7-595e-4895-bcd4-5fec9c9d78e9","Type":"ContainerDied","Data":"a6ef06136594319e95049c1f8719e92eb131384c1ba6f735997606f88a6195c4"} Oct 30 01:12:43.255879 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.255462 5300 generic.go:356] "Generic (PLEG): container finished" podID="7bd004c0-3447-4382-82f5-e797f86b850e" containerID="017ed38f55ce4b5a817d8e5863c39dafb727cc66d33f24b33d6bb300359b470b" exitCode=0 Oct 30 01:12:43.255879 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.255709 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" event={"ID":"7bd004c0-3447-4382-82f5-e797f86b850e","Type":"ContainerDied","Data":"017ed38f55ce4b5a817d8e5863c39dafb727cc66d33f24b33d6bb300359b470b"} Oct 30 01:12:43.255879 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.255789 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" event={"ID":"7bd004c0-3447-4382-82f5-e797f86b850e","Type":"ContainerDied","Data":"8a7603eda0b57f931831c65f8db5ad23ea8715afcd55126b5eafc0576a5737f6"} Oct 30 01:12:43.255879 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.255817 5300 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a7603eda0b57f931831c65f8db5ad23ea8715afcd55126b5eafc0576a5737f6" Oct 30 01:12:43.265231 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.265204 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:12:43.295899 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.295424 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bd004c0-3447-4382-82f5-e797f86b850e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd004c0-3447-4382-82f5-e797f86b850e-istiod-ca-cert\") pod \"7bd004c0-3447-4382-82f5-e797f86b850e\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " Oct 30 01:12:43.296306 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.296272 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-istio-data\") pod \"7bd004c0-3447-4382-82f5-e797f86b850e\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " Oct 30 01:12:43.296474 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.296376 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bd004c0-3447-4382-82f5-e797f86b850e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd004c0-3447-4382-82f5-e797f86b850e-istio-podinfo\") pod \"7bd004c0-3447-4382-82f5-e797f86b850e\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " Oct 30 01:12:43.296474 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.296437 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-workload-certs\") pod \"7bd004c0-3447-4382-82f5-e797f86b850e\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " Oct 30 01:12:43.296678 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.296488 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-workload-socket\") pod \"7bd004c0-3447-4382-82f5-e797f86b850e\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " Oct 30 01:12:43.296678 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.296542 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-credential-socket\") pod \"7bd004c0-3447-4382-82f5-e797f86b850e\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " Oct 30 01:12:43.296678 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.296574 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd004c0-3447-4382-82f5-e797f86b850e-kube-api-access-p9tnd\" (UniqueName: \"kubernetes.io/projected/7bd004c0-3447-4382-82f5-e797f86b850e-kube-api-access-p9tnd\") pod \"7bd004c0-3447-4382-82f5-e797f86b850e\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " Oct 30 01:12:43.296678 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.296642 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-istio-envoy\") pod \"7bd004c0-3447-4382-82f5-e797f86b850e\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " Oct 30 01:12:43.297063 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.296780 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd004c0-3447-4382-82f5-e797f86b850e-istio-token\" (UniqueName: \"kubernetes.io/projected/7bd004c0-3447-4382-82f5-e797f86b850e-istio-token\") pod \"7bd004c0-3447-4382-82f5-e797f86b850e\" (UID: \"7bd004c0-3447-4382-82f5-e797f86b850e\") " Oct 30 01:12:43.297063 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.295867 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd004c0-3447-4382-82f5-e797f86b850e-istiod-ca-cert" pod "7bd004c0-3447-4382-82f5-e797f86b850e" (UID: "7bd004c0-3447-4382-82f5-e797f86b850e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:12:43.297063 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.296998 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-workload-certs" pod "7bd004c0-3447-4382-82f5-e797f86b850e" (UID: "7bd004c0-3447-4382-82f5-e797f86b850e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:43.297474 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.297202 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-istio-data" pod "7bd004c0-3447-4382-82f5-e797f86b850e" (UID: "7bd004c0-3447-4382-82f5-e797f86b850e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:43.297745 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.297720 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-credential-socket" pod "7bd004c0-3447-4382-82f5-e797f86b850e" (UID: "7bd004c0-3447-4382-82f5-e797f86b850e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:43.297967 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.297948 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-workload-socket" pod "7bd004c0-3447-4382-82f5-e797f86b850e" (UID: "7bd004c0-3447-4382-82f5-e797f86b850e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:43.308096 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.301606 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bd004c0-3447-4382-82f5-e797f86b850e-istio-podinfo" pod "7bd004c0-3447-4382-82f5-e797f86b850e" (UID: "7bd004c0-3447-4382-82f5-e797f86b850e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:12:43.325805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.325734 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd004c0-3447-4382-82f5-e797f86b850e-kube-api-access-p9tnd" pod "7bd004c0-3447-4382-82f5-e797f86b850e" (UID: "7bd004c0-3447-4382-82f5-e797f86b850e"). InnerVolumeSpecName "kube-api-access-p9tnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:43.328959 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.328928 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.329146 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.329132 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.329242 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.329230 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.329329 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.329318 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9tnd\" (UniqueName: \"kubernetes.io/projected/7bd004c0-3447-4382-82f5-e797f86b850e-kube-api-access-p9tnd\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.329417 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.329407 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd004c0-3447-4382-82f5-e797f86b850e-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.329498 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.329488 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.329583 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.329573 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd004c0-3447-4382-82f5-e797f86b850e-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.346789 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.346737 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:43.361170 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.361111 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-istio-envoy" pod "7bd004c0-3447-4382-82f5-e797f86b850e" (UID: "7bd004c0-3447-4382-82f5-e797f86b850e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:43.368342 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.368290 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd004c0-3447-4382-82f5-e797f86b850e-istio-token" pod "7bd004c0-3447-4382-82f5-e797f86b850e" (UID: "7bd004c0-3447-4382-82f5-e797f86b850e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:43.433019 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.432964 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd004c0-3447-4382-82f5-e797f86b850e-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.433347 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.433310 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd004c0-3447-4382-82f5-e797f86b850e-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.483421 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.483383 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:12:43.539399 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.539360 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-token\" (UniqueName: \"kubernetes.io/projected/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-token\") pod \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " Oct 30 01:12:43.539568 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.539441 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-credential-socket\") pod \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " Oct 30 01:12:43.539568 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.539503 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istiod-ca-cert\") pod \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " Oct 30 01:12:43.539704 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.539590 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-podinfo\") pod \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " Oct 30 01:12:43.539704 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.539645 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-envoy\") pod \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " Oct 30 01:12:43.539852 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.539724 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-data\") pod \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " Oct 30 01:12:43.539887 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.539860 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-kube-api-access-tmlll\" (UniqueName: \"kubernetes.io/projected/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-kube-api-access-tmlll\") pod \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " Oct 30 01:12:43.539949 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.539932 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-workload-socket\") pod \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " Oct 30 01:12:43.540019 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.540005 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-workload-certs\") pod \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\" (UID: \"22aacbb7-595e-4895-bcd4-5fec9c9d78e9\") " Oct 30 01:12:43.541453 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.541422 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-data" pod "22aacbb7-595e-4895-bcd4-5fec9c9d78e9" (UID: "22aacbb7-595e-4895-bcd4-5fec9c9d78e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:43.541818 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.541794 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-credential-socket" pod "22aacbb7-595e-4895-bcd4-5fec9c9d78e9" (UID: "22aacbb7-595e-4895-bcd4-5fec9c9d78e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:43.542439 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.542404 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-workload-socket" pod "22aacbb7-595e-4895-bcd4-5fec9c9d78e9" (UID: "22aacbb7-595e-4895-bcd4-5fec9c9d78e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:43.542696 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.542653 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-workload-certs" pod "22aacbb7-595e-4895-bcd4-5fec9c9d78e9" (UID: "22aacbb7-595e-4895-bcd4-5fec9c9d78e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:43.544558 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.544517 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-kube-api-access-tmlll" pod "22aacbb7-595e-4895-bcd4-5fec9c9d78e9" (UID: "22aacbb7-595e-4895-bcd4-5fec9c9d78e9"). InnerVolumeSpecName "kube-api-access-tmlll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:43.547411 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.546901 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-envoy" pod "22aacbb7-595e-4895-bcd4-5fec9c9d78e9" (UID: "22aacbb7-595e-4895-bcd4-5fec9c9d78e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:43.554240 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.554188 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istiod-ca-cert" pod "22aacbb7-595e-4895-bcd4-5fec9c9d78e9" (UID: "22aacbb7-595e-4895-bcd4-5fec9c9d78e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:12:43.564221 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.564159 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-token" pod "22aacbb7-595e-4895-bcd4-5fec9c9d78e9" (UID: "22aacbb7-595e-4895-bcd4-5fec9c9d78e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:43.565551 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.565504 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-podinfo" pod "22aacbb7-595e-4895-bcd4-5fec9c9d78e9" (UID: "22aacbb7-595e-4895-bcd4-5fec9c9d78e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:12:43.643659 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.643474 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.643659 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.643513 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.643659 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.643528 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.643659 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.643543 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.643659 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.643557 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmlll\" (UniqueName: \"kubernetes.io/projected/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-kube-api-access-tmlll\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.643659 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.643572 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.643659 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.643585 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.643659 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.643598 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.643659 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.643611 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22aacbb7-595e-4895-bcd4-5fec9c9d78e9-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.784054 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.784027 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 01:12:43.845830 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.845797 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-data\") pod \"59536065-cfa0-4f0c-a2ca-55284c369d81\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " Oct 30 01:12:43.846129 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.846107 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59536065-cfa0-4f0c-a2ca-55284c369d81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59536065-cfa0-4f0c-a2ca-55284c369d81-istiod-ca-cert\") pod \"59536065-cfa0-4f0c-a2ca-55284c369d81\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " Oct 30 01:12:43.846290 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.846273 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-workload-certs\") pod \"59536065-cfa0-4f0c-a2ca-55284c369d81\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " Oct 30 01:12:43.846533 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.846514 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-workload-socket\") pod \"59536065-cfa0-4f0c-a2ca-55284c369d81\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " Oct 30 01:12:43.846743 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.846715 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-workload-socket" pod "59536065-cfa0-4f0c-a2ca-55284c369d81" (UID: "59536065-cfa0-4f0c-a2ca-55284c369d81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:43.846954 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.846737 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59536065-cfa0-4f0c-a2ca-55284c369d81-istiod-ca-cert" pod "59536065-cfa0-4f0c-a2ca-55284c369d81" (UID: "59536065-cfa0-4f0c-a2ca-55284c369d81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:12:43.847195 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.847160 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-workload-certs" pod "59536065-cfa0-4f0c-a2ca-55284c369d81" (UID: "59536065-cfa0-4f0c-a2ca-55284c369d81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:43.847323 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.847245 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-envoy\") pod \"59536065-cfa0-4f0c-a2ca-55284c369d81\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " Oct 30 01:12:43.847601 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.847573 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-credential-socket\") pod \"59536065-cfa0-4f0c-a2ca-55284c369d81\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " Oct 30 01:12:43.847875 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.847850 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-credential-socket" pod "59536065-cfa0-4f0c-a2ca-55284c369d81" (UID: "59536065-cfa0-4f0c-a2ca-55284c369d81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:43.848010 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.847987 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59536065-cfa0-4f0c-a2ca-55284c369d81-kube-api-access-7vdrx\" (UniqueName: \"kubernetes.io/projected/59536065-cfa0-4f0c-a2ca-55284c369d81-kube-api-access-7vdrx\") pod \"59536065-cfa0-4f0c-a2ca-55284c369d81\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " Oct 30 01:12:43.848341 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.848321 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-podinfo\") pod \"59536065-cfa0-4f0c-a2ca-55284c369d81\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " Oct 30 01:12:43.848503 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.848487 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-token\" (UniqueName: \"kubernetes.io/projected/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-token\") pod \"59536065-cfa0-4f0c-a2ca-55284c369d81\" (UID: \"59536065-cfa0-4f0c-a2ca-55284c369d81\") " Oct 30 01:12:43.849391 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.849367 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59536065-cfa0-4f0c-a2ca-55284c369d81-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.849875 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.849849 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.849999 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.849986 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.850112 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.850087 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.851198 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.851175 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-envoy" pod "59536065-cfa0-4f0c-a2ca-55284c369d81" (UID: "59536065-cfa0-4f0c-a2ca-55284c369d81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:43.852304 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.852274 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-data" pod "59536065-cfa0-4f0c-a2ca-55284c369d81" (UID: "59536065-cfa0-4f0c-a2ca-55284c369d81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:12:43.861024 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.860989 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-podinfo" pod "59536065-cfa0-4f0c-a2ca-55284c369d81" (UID: "59536065-cfa0-4f0c-a2ca-55284c369d81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:12:43.861220 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.861185 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59536065-cfa0-4f0c-a2ca-55284c369d81-kube-api-access-7vdrx" pod "59536065-cfa0-4f0c-a2ca-55284c369d81" (UID: "59536065-cfa0-4f0c-a2ca-55284c369d81"). InnerVolumeSpecName "kube-api-access-7vdrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:43.861348 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.861328 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-token" pod "59536065-cfa0-4f0c-a2ca-55284c369d81" (UID: "59536065-cfa0-4f0c-a2ca-55284c369d81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:12:43.953499 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.953455 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vdrx\" (UniqueName: \"kubernetes.io/projected/59536065-cfa0-4f0c-a2ca-55284c369d81-kube-api-access-7vdrx\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.953499 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.953500 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.953777 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.953514 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.953777 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.953526 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:43.953777 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:43.953538 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59536065-cfa0-4f0c-a2ca-55284c369d81-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:12:44.264631 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:44.264596 5300 generic.go:356] "Generic (PLEG): container finished" podID="59536065-cfa0-4f0c-a2ca-55284c369d81" containerID="42661b1ed5c2d29b94d756ac672ebbaf0990f6d22d30b33c09dbf42ddf99c06f" exitCode=0 Oct 30 01:12:44.264837 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:44.264666 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" Oct 30 01:12:44.264837 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:44.264678 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" event={"ID":"59536065-cfa0-4f0c-a2ca-55284c369d81","Type":"ContainerDied","Data":"42661b1ed5c2d29b94d756ac672ebbaf0990f6d22d30b33c09dbf42ddf99c06f"} Oct 30 01:12:44.264837 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:44.264742 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2" event={"ID":"59536065-cfa0-4f0c-a2ca-55284c369d81","Type":"ContainerDied","Data":"19307bb9bb200a86d3bef422e2c602fa6abe6425f04e74d0b82d65792123d95a"} Oct 30 01:12:44.264837 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:44.264788 5300 scope.go:122] "RemoveContainer" containerID="42661b1ed5c2d29b94d756ac672ebbaf0990f6d22d30b33c09dbf42ddf99c06f" Oct 30 01:12:44.266327 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:44.266305 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" Oct 30 01:12:44.266327 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:44.266317 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn" event={"ID":"22aacbb7-595e-4895-bcd4-5fec9c9d78e9","Type":"ContainerDied","Data":"4b6dd67d891c5eac6e7d1d4ab6bc9bdc3abfab7cd02e3adc9a751968646bbf96"} Oct 30 01:12:44.266508 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:44.266474 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm" Oct 30 01:12:44.277036 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:44.277015 5300 scope.go:122] "RemoveContainer" containerID="42661b1ed5c2d29b94d756ac672ebbaf0990f6d22d30b33c09dbf42ddf99c06f" Oct 30 01:12:44.277336 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:12:44.277317 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42661b1ed5c2d29b94d756ac672ebbaf0990f6d22d30b33c09dbf42ddf99c06f\": container with ID starting with 42661b1ed5c2d29b94d756ac672ebbaf0990f6d22d30b33c09dbf42ddf99c06f not found: ID does not exist" containerID="42661b1ed5c2d29b94d756ac672ebbaf0990f6d22d30b33c09dbf42ddf99c06f" Oct 30 01:12:44.277409 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:44.277348 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42661b1ed5c2d29b94d756ac672ebbaf0990f6d22d30b33c09dbf42ddf99c06f"} err="failed to get container status \"42661b1ed5c2d29b94d756ac672ebbaf0990f6d22d30b33c09dbf42ddf99c06f\": rpc error: code = NotFound desc = could not find container \"42661b1ed5c2d29b94d756ac672ebbaf0990f6d22d30b33c09dbf42ddf99c06f\": container with ID starting with 42661b1ed5c2d29b94d756ac672ebbaf0990f6d22d30b33c09dbf42ddf99c06f not found: ID does not exist" Oct 30 01:12:44.277409 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:44.277372 5300 scope.go:122] "RemoveContainer" containerID="a6ef06136594319e95049c1f8719e92eb131384c1ba6f735997606f88a6195c4" Oct 30 01:12:44.326041 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:44.325996 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2"] Oct 30 01:12:44.338042 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:44.338003 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-vhvt2"] Oct 30 01:12:44.367071 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:44.367028 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn"] Oct 30 01:12:44.377458 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:44.377423 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-82tzn"] Oct 30 01:12:44.416844 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:44.416810 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm"] Oct 30 01:12:44.422589 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:44.422560 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-7vlzm"] Oct 30 01:12:44.607061 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:44.606973 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:44.706692 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:44.706650 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:45.826680 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:45.826646 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22aacbb7-595e-4895-bcd4-5fec9c9d78e9" path="/var/lib/kubelet/pods/22aacbb7-595e-4895-bcd4-5fec9c9d78e9/volumes" Oct 30 01:12:45.827250 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:45.827234 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59536065-cfa0-4f0c-a2ca-55284c369d81" path="/var/lib/kubelet/pods/59536065-cfa0-4f0c-a2ca-55284c369d81/volumes" Oct 30 01:12:45.827685 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:45.827672 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bd004c0-3447-4382-82f5-e797f86b850e" path="/var/lib/kubelet/pods/7bd004c0-3447-4382-82f5-e797f86b850e/volumes" Oct 30 01:12:46.553495 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.553459 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65"] Oct 30 01:12:46.554287 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.554270 5300 cpu_manager.go:414] "containerMap: removing container" podUID="22aacbb7-595e-4895-bcd4-5fec9c9d78e9" containerName="istio-proxy" Oct 30 01:12:46.554371 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.554288 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22aacbb7-595e-4895-bcd4-5fec9c9d78e9" containerName="istio-proxy" Oct 30 01:12:46.554371 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.554310 5300 cpu_manager.go:414] "containerMap: removing container" podUID="7bd004c0-3447-4382-82f5-e797f86b850e" containerName="istio-proxy" Oct 30 01:12:46.554371 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.554315 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd004c0-3447-4382-82f5-e797f86b850e" containerName="istio-proxy" Oct 30 01:12:46.554371 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.554333 5300 cpu_manager.go:414] "containerMap: removing container" podUID="59536065-cfa0-4f0c-a2ca-55284c369d81" containerName="istio-proxy" Oct 30 01:12:46.554371 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.554341 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59536065-cfa0-4f0c-a2ca-55284c369d81" containerName="istio-proxy" Oct 30 01:12:46.554608 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.554459 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="22aacbb7-595e-4895-bcd4-5fec9c9d78e9" containerName="istio-proxy" Oct 30 01:12:46.554608 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.554471 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd004c0-3447-4382-82f5-e797f86b850e" containerName="istio-proxy" Oct 30 01:12:46.554608 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.554481 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="59536065-cfa0-4f0c-a2ca-55284c369d81" containerName="istio-proxy" Oct 30 01:12:46.563642 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.563600 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65" Oct 30 01:12:46.581608 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.581574 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65"] Oct 30 01:12:46.590159 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.590123 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-34e896-serving-cert\"" Oct 30 01:12:46.697737 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.697690 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v79kg\" (UniqueName: \"kubernetes.io/projected/f855c133-b20f-4baa-981e-55345624598f-kube-api-access-v79kg\") pod \"maas-api-e2e-inf-b-34e896-565d8c884c-sjd65\" (UID: \"f855c133-b20f-4baa-981e-55345624598f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65" Oct 30 01:12:46.698034 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.698013 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f855c133-b20f-4baa-981e-55345624598f-maas-api-tls\") pod \"maas-api-e2e-inf-b-34e896-565d8c884c-sjd65\" (UID: \"f855c133-b20f-4baa-981e-55345624598f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65" Oct 30 01:12:46.698125 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.698054 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f855c133-b20f-4baa-981e-55345624598f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-34e896-565d8c884c-sjd65\" (UID: \"f855c133-b20f-4baa-981e-55345624598f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65" Oct 30 01:12:46.698192 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.698168 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f855c133-b20f-4baa-981e-55345624598f-metrics-certs\") pod \"maas-api-e2e-inf-b-34e896-565d8c884c-sjd65\" (UID: \"f855c133-b20f-4baa-981e-55345624598f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65" Oct 30 01:12:46.727310 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.727270 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:46.799847 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.799807 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f855c133-b20f-4baa-981e-55345624598f-maas-api-tls\") pod \"maas-api-e2e-inf-b-34e896-565d8c884c-sjd65\" (UID: \"f855c133-b20f-4baa-981e-55345624598f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65" Oct 30 01:12:46.800037 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.799864 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f855c133-b20f-4baa-981e-55345624598f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-34e896-565d8c884c-sjd65\" (UID: \"f855c133-b20f-4baa-981e-55345624598f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65" Oct 30 01:12:46.800037 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.799911 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f855c133-b20f-4baa-981e-55345624598f-metrics-certs\") pod \"maas-api-e2e-inf-b-34e896-565d8c884c-sjd65\" (UID: \"f855c133-b20f-4baa-981e-55345624598f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65" Oct 30 01:12:46.800037 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.800022 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v79kg\" (UniqueName: \"kubernetes.io/projected/f855c133-b20f-4baa-981e-55345624598f-kube-api-access-v79kg\") pod \"maas-api-e2e-inf-b-34e896-565d8c884c-sjd65\" (UID: \"f855c133-b20f-4baa-981e-55345624598f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65" Oct 30 01:12:46.800537 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.800508 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f855c133-b20f-4baa-981e-55345624598f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-34e896-565d8c884c-sjd65\" (UID: \"f855c133-b20f-4baa-981e-55345624598f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65" Oct 30 01:12:46.802548 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.802527 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f855c133-b20f-4baa-981e-55345624598f-metrics-certs\") pod \"maas-api-e2e-inf-b-34e896-565d8c884c-sjd65\" (UID: \"f855c133-b20f-4baa-981e-55345624598f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65" Oct 30 01:12:46.802548 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.802539 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f855c133-b20f-4baa-981e-55345624598f-maas-api-tls\") pod \"maas-api-e2e-inf-b-34e896-565d8c884c-sjd65\" (UID: \"f855c133-b20f-4baa-981e-55345624598f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65" Oct 30 01:12:46.821249 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.821186 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v79kg\" (UniqueName: \"kubernetes.io/projected/f855c133-b20f-4baa-981e-55345624598f-kube-api-access-v79kg\") pod \"maas-api-e2e-inf-b-34e896-565d8c884c-sjd65\" (UID: \"f855c133-b20f-4baa-981e-55345624598f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65" Oct 30 01:12:46.849042 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.849009 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:46.885847 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:46.884022 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65" Oct 30 01:12:47.199743 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:47.199708 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65"] Oct 30 01:12:47.199875 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:12:47.199799 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf855c133_b20f_4baa_981e_55345624598f.slice/crio-6a1ccfdeb6f5dec0fb9b3b1451be1948ba5f3b76ef4bd96c30cc571aeb06bc9c WatchSource:0}: Error finding container 6a1ccfdeb6f5dec0fb9b3b1451be1948ba5f3b76ef4bd96c30cc571aeb06bc9c: Status 404 returned error can't find the container with id 6a1ccfdeb6f5dec0fb9b3b1451be1948ba5f3b76ef4bd96c30cc571aeb06bc9c Oct 30 01:12:47.201219 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:47.201199 5300 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:12:47.342157 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:47.342121 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65" event={"ID":"f855c133-b20f-4baa-981e-55345624598f","Type":"ContainerStarted","Data":"6a1ccfdeb6f5dec0fb9b3b1451be1948ba5f3b76ef4bd96c30cc571aeb06bc9c"} Oct 30 01:12:48.351504 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:48.351465 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65" event={"ID":"f855c133-b20f-4baa-981e-55345624598f","Type":"ContainerStarted","Data":"b93e8be534f921bff7973500d894d4975577a97da363591c39b40c22dde59084"} Oct 30 01:12:48.352048 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:48.351529 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65" Oct 30 01:12:48.398878 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:48.398822 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65" podStartSLOduration=2.070532075 podStartE2EDuration="2.398806439s" totalImagesPullingTime="328.274364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:48.395322135 +0000 UTC m=+1768.711190460" watchObservedRunningTime="2026-10-30 01:12:48.398806439 +0000 UTC m=+1768.714674763" Oct 30 01:12:48.562423 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:48.562380 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:50.143273 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:50.143236 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:51.594088 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:12:51.594050 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:12:51.913154 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:51.913063 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:53.576298 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:53.576257 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:54.364363 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:54.364333 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65" Oct 30 01:12:55.359893 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:55.359852 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:57.179054 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:57.179022 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:58.296753 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.296715 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4"] Oct 30 01:12:58.304862 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.304833 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" Oct 30 01:12:58.322693 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.322648 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4"] Oct 30 01:12:58.330408 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.330379 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-687471\"/\"openshift-service-ca.crt\"" Oct 30 01:12:58.330583 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.330562 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-687471\"/\"test-model-687471-kserve-self-signed-certs\"" Oct 30 01:12:58.336394 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.336365 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-687471\"/\"default-dockercfg-j64ff\"" Oct 30 01:12:58.336394 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.336364 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-687471\"/\"kube-root-ca.crt\"" Oct 30 01:12:58.356976 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.356942 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:58.389626 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.389594 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66f98a24-bdba-4445-be54-6e30550e21da-tls-certs\") pod \"test-model-687471-kserve-65dc8ff97c-r84r4\" (UID: \"66f98a24-bdba-4445-be54-6e30550e21da\") " pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" Oct 30 01:12:58.389841 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.389662 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-tmp-dir\") pod \"test-model-687471-kserve-65dc8ff97c-r84r4\" (UID: \"66f98a24-bdba-4445-be54-6e30550e21da\") " pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" Oct 30 01:12:58.389841 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.389819 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85fsc\" (UniqueName: \"kubernetes.io/projected/66f98a24-bdba-4445-be54-6e30550e21da-kube-api-access-85fsc\") pod \"test-model-687471-kserve-65dc8ff97c-r84r4\" (UID: \"66f98a24-bdba-4445-be54-6e30550e21da\") " pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" Oct 30 01:12:58.390053 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.390031 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-model-cache\") pod \"test-model-687471-kserve-65dc8ff97c-r84r4\" (UID: \"66f98a24-bdba-4445-be54-6e30550e21da\") " pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" Oct 30 01:12:58.390183 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.390158 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-home\") pod \"test-model-687471-kserve-65dc8ff97c-r84r4\" (UID: \"66f98a24-bdba-4445-be54-6e30550e21da\") " pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" Oct 30 01:12:58.390283 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.390266 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-dshm\") pod \"test-model-687471-kserve-65dc8ff97c-r84r4\" (UID: \"66f98a24-bdba-4445-be54-6e30550e21da\") " pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" Oct 30 01:12:58.491909 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.491876 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-home\") pod \"test-model-687471-kserve-65dc8ff97c-r84r4\" (UID: \"66f98a24-bdba-4445-be54-6e30550e21da\") " pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" Oct 30 01:12:58.492083 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.491944 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-dshm\") pod \"test-model-687471-kserve-65dc8ff97c-r84r4\" (UID: \"66f98a24-bdba-4445-be54-6e30550e21da\") " pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" Oct 30 01:12:58.492083 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.491999 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66f98a24-bdba-4445-be54-6e30550e21da-tls-certs\") pod \"test-model-687471-kserve-65dc8ff97c-r84r4\" (UID: \"66f98a24-bdba-4445-be54-6e30550e21da\") " pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" Oct 30 01:12:58.492083 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.492041 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-tmp-dir\") pod \"test-model-687471-kserve-65dc8ff97c-r84r4\" (UID: \"66f98a24-bdba-4445-be54-6e30550e21da\") " pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" Oct 30 01:12:58.492224 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.492090 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85fsc\" (UniqueName: \"kubernetes.io/projected/66f98a24-bdba-4445-be54-6e30550e21da-kube-api-access-85fsc\") pod \"test-model-687471-kserve-65dc8ff97c-r84r4\" (UID: \"66f98a24-bdba-4445-be54-6e30550e21da\") " pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" Oct 30 01:12:58.492284 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.492254 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-model-cache\") pod \"test-model-687471-kserve-65dc8ff97c-r84r4\" (UID: \"66f98a24-bdba-4445-be54-6e30550e21da\") " pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" Oct 30 01:12:58.492336 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.492315 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-home\") pod \"test-model-687471-kserve-65dc8ff97c-r84r4\" (UID: \"66f98a24-bdba-4445-be54-6e30550e21da\") " pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" Oct 30 01:12:58.492463 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.492427 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-tmp-dir\") pod \"test-model-687471-kserve-65dc8ff97c-r84r4\" (UID: \"66f98a24-bdba-4445-be54-6e30550e21da\") " pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" Oct 30 01:12:58.492579 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.492508 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-model-cache\") pod \"test-model-687471-kserve-65dc8ff97c-r84r4\" (UID: \"66f98a24-bdba-4445-be54-6e30550e21da\") " pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" Oct 30 01:12:58.494546 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.494519 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-dshm\") pod \"test-model-687471-kserve-65dc8ff97c-r84r4\" (UID: \"66f98a24-bdba-4445-be54-6e30550e21da\") " pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" Oct 30 01:12:58.494872 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.494855 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66f98a24-bdba-4445-be54-6e30550e21da-tls-certs\") pod \"test-model-687471-kserve-65dc8ff97c-r84r4\" (UID: \"66f98a24-bdba-4445-be54-6e30550e21da\") " pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" Oct 30 01:12:58.514129 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.514100 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85fsc\" (UniqueName: \"kubernetes.io/projected/66f98a24-bdba-4445-be54-6e30550e21da-kube-api-access-85fsc\") pod \"test-model-687471-kserve-65dc8ff97c-r84r4\" (UID: \"66f98a24-bdba-4445-be54-6e30550e21da\") " pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" Oct 30 01:12:58.627333 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.626973 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" Oct 30 01:12:58.943280 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:58.943240 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4"] Oct 30 01:12:58.944359 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:12:58.944334 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66f98a24_bdba_4445_be54_6e30550e21da.slice/crio-7b72dac311f7aa5d4fab33065c24f39ab23260a89d9095345c8891ed6c112174 WatchSource:0}: Error finding container 7b72dac311f7aa5d4fab33065c24f39ab23260a89d9095345c8891ed6c112174: Status 404 returned error can't find the container with id 7b72dac311f7aa5d4fab33065c24f39ab23260a89d9095345c8891ed6c112174 Oct 30 01:12:59.094154 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:59.094119 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:12:59.453395 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:59.453355 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" event={"ID":"66f98a24-bdba-4445-be54-6e30550e21da","Type":"ContainerStarted","Data":"0a90f9d9ffc43fb64019dcdf33d304ea6c6fba5e71042dcefc1a7da75a54d7b2"} Oct 30 01:12:59.453842 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:59.453406 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" event={"ID":"66f98a24-bdba-4445-be54-6e30550e21da","Type":"ContainerStarted","Data":"7b72dac311f7aa5d4fab33065c24f39ab23260a89d9095345c8891ed6c112174"} Oct 30 01:12:59.497940 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:59.497883 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" podStartSLOduration=1.497861578 podStartE2EDuration="1.497861578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:59.496930127 +0000 UTC m=+1779.812798452" watchObservedRunningTime="2026-10-30 01:12:59.497861578 +0000 UTC m=+1779.813729955" Oct 30 01:12:59.628111 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:12:59.628078 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" Oct 30 01:13:00.101321 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:00.101285 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:00.207638 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:00.207603 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:00.775514 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:00.775475 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:01.795299 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:01.795251 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:02.565161 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:02.565129 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:02.675120 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:02.675089 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:03.063556 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.063521 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk"] Oct 30 01:13:03.068412 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.068380 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" Oct 30 01:13:03.105476 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.105443 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e92dab\"/\"default-dockercfg-sbv7c\"" Oct 30 01:13:03.105724 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.105459 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e92dab\"/\"kube-root-ca.crt\"" Oct 30 01:13:03.105724 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.105516 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e92dab\"/\"openshift-service-ca.crt\"" Oct 30 01:13:03.105963 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.105529 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e92dab\"/\"test-model-e92dab-kserve-self-signed-certs\"" Oct 30 01:13:03.108668 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.108638 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:03.110273 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.110251 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk"] Oct 30 01:13:03.173052 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.173013 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-dshm\") pod \"test-model-e92dab-kserve-686fcb5c77-xcprk\" (UID: \"26f9928d-0276-4c79-800a-9886a99c6499\") " pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" Oct 30 01:13:03.173240 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.173147 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26f9928d-0276-4c79-800a-9886a99c6499-tls-certs\") pod \"test-model-e92dab-kserve-686fcb5c77-xcprk\" (UID: \"26f9928d-0276-4c79-800a-9886a99c6499\") " pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" Oct 30 01:13:03.173240 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.173226 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cctx\" (UniqueName: \"kubernetes.io/projected/26f9928d-0276-4c79-800a-9886a99c6499-kube-api-access-5cctx\") pod \"test-model-e92dab-kserve-686fcb5c77-xcprk\" (UID: \"26f9928d-0276-4c79-800a-9886a99c6499\") " pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" Oct 30 01:13:03.173323 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.173268 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-tmp-dir\") pod \"test-model-e92dab-kserve-686fcb5c77-xcprk\" (UID: \"26f9928d-0276-4c79-800a-9886a99c6499\") " pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" Oct 30 01:13:03.173323 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.173304 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-home\") pod \"test-model-e92dab-kserve-686fcb5c77-xcprk\" (UID: \"26f9928d-0276-4c79-800a-9886a99c6499\") " pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" Oct 30 01:13:03.173458 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.173443 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-model-cache\") pod \"test-model-e92dab-kserve-686fcb5c77-xcprk\" (UID: \"26f9928d-0276-4c79-800a-9886a99c6499\") " pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" Oct 30 01:13:03.284821 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.281066 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-dshm\") pod \"test-model-e92dab-kserve-686fcb5c77-xcprk\" (UID: \"26f9928d-0276-4c79-800a-9886a99c6499\") " pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" Oct 30 01:13:03.284821 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.281225 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26f9928d-0276-4c79-800a-9886a99c6499-tls-certs\") pod \"test-model-e92dab-kserve-686fcb5c77-xcprk\" (UID: \"26f9928d-0276-4c79-800a-9886a99c6499\") " pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" Oct 30 01:13:03.284821 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.281313 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cctx\" (UniqueName: \"kubernetes.io/projected/26f9928d-0276-4c79-800a-9886a99c6499-kube-api-access-5cctx\") pod \"test-model-e92dab-kserve-686fcb5c77-xcprk\" (UID: \"26f9928d-0276-4c79-800a-9886a99c6499\") " pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" Oct 30 01:13:03.284821 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.281376 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-tmp-dir\") pod \"test-model-e92dab-kserve-686fcb5c77-xcprk\" (UID: \"26f9928d-0276-4c79-800a-9886a99c6499\") " pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" Oct 30 01:13:03.284821 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.281422 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-home\") pod \"test-model-e92dab-kserve-686fcb5c77-xcprk\" (UID: \"26f9928d-0276-4c79-800a-9886a99c6499\") " pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" Oct 30 01:13:03.284821 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.281529 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-model-cache\") pod \"test-model-e92dab-kserve-686fcb5c77-xcprk\" (UID: \"26f9928d-0276-4c79-800a-9886a99c6499\") " pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" Oct 30 01:13:03.284821 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.282275 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-model-cache\") pod \"test-model-e92dab-kserve-686fcb5c77-xcprk\" (UID: \"26f9928d-0276-4c79-800a-9886a99c6499\") " pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" Oct 30 01:13:03.284821 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.283153 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-tmp-dir\") pod \"test-model-e92dab-kserve-686fcb5c77-xcprk\" (UID: \"26f9928d-0276-4c79-800a-9886a99c6499\") " pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" Oct 30 01:13:03.284821 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.283410 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-home\") pod \"test-model-e92dab-kserve-686fcb5c77-xcprk\" (UID: \"26f9928d-0276-4c79-800a-9886a99c6499\") " pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" Oct 30 01:13:03.285420 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.285340 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-dshm\") pod \"test-model-e92dab-kserve-686fcb5c77-xcprk\" (UID: \"26f9928d-0276-4c79-800a-9886a99c6499\") " pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" Oct 30 01:13:03.294318 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.294283 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26f9928d-0276-4c79-800a-9886a99c6499-tls-certs\") pod \"test-model-e92dab-kserve-686fcb5c77-xcprk\" (UID: \"26f9928d-0276-4c79-800a-9886a99c6499\") " pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" Oct 30 01:13:03.311931 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.311880 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cctx\" (UniqueName: \"kubernetes.io/projected/26f9928d-0276-4c79-800a-9886a99c6499-kube-api-access-5cctx\") pod \"test-model-e92dab-kserve-686fcb5c77-xcprk\" (UID: \"26f9928d-0276-4c79-800a-9886a99c6499\") " pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" Oct 30 01:13:03.388388 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.388295 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" Oct 30 01:13:03.734811 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:03.734781 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk"] Oct 30 01:13:03.735136 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:13:03.735110 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26f9928d_0276_4c79_800a_9886a99c6499.slice/crio-acac3317495897580373d4b82410bc2dcae1abbbb1f5b02f7b4bb889dc84a482 WatchSource:0}: Error finding container acac3317495897580373d4b82410bc2dcae1abbbb1f5b02f7b4bb889dc84a482: Status 404 returned error can't find the container with id acac3317495897580373d4b82410bc2dcae1abbbb1f5b02f7b4bb889dc84a482 Oct 30 01:13:04.380951 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:04.380896 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:04.498941 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:04.498901 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" event={"ID":"26f9928d-0276-4c79-800a-9886a99c6499","Type":"ContainerStarted","Data":"e86d4632e9eb84c2bbb3373c31c5d5bce2adaa8155992197eeb4ddb0c13409f1"} Oct 30 01:13:04.498941 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:04.498955 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" event={"ID":"26f9928d-0276-4c79-800a-9886a99c6499","Type":"ContainerStarted","Data":"acac3317495897580373d4b82410bc2dcae1abbbb1f5b02f7b4bb889dc84a482"} Oct 30 01:13:04.542871 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:04.542820 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" podStartSLOduration=1.5428062439999999 podStartE2EDuration="1.542806244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:04.540451863 +0000 UTC m=+1784.856320187" watchObservedRunningTime="2026-10-30 01:13:04.542806244 +0000 UTC m=+1784.858674568" Oct 30 01:13:05.389389 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:05.389347 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" Oct 30 01:13:05.974075 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:05.974044 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:07.412985 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:07.412952 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:07.948228 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:07.948158 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:08.058386 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:08.058347 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:08.161454 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:08.161418 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:08.562016 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:08.561982 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:08.629682 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:08.629632 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" Oct 30 01:13:08.641234 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:08.641209 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" Oct 30 01:13:09.590349 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:09.590312 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" Oct 30 01:13:10.144023 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:10.143966 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:11.957686 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:11.957648 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:13.389790 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.389723 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" Oct 30 01:13:13.403035 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.403006 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" Oct 30 01:13:13.615262 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.615231 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" Oct 30 01:13:13.693528 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.693437 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:13.702045 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.701886 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s"] Oct 30 01:13:13.716815 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.712015 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s"] Oct 30 01:13:13.716815 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.712138 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" Oct 30 01:13:13.749200 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.749165 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-56d1d9\"/\"kube-root-ca.crt\"" Oct 30 01:13:13.750382 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.750357 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-56d1d9\"/\"default-dockercfg-5vw6r\"" Oct 30 01:13:13.750666 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.750646 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-56d1d9\"/\"test-model-56d1d9-kserve-self-signed-certs\"" Oct 30 01:13:13.750933 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.750916 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-56d1d9\"/\"openshift-service-ca.crt\"" Oct 30 01:13:13.874005 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.873848 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-model-cache\") pod \"test-model-56d1d9-kserve-65d4df9bdf-7n97s\" (UID: \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\") " pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" Oct 30 01:13:13.884881 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.884844 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e84bfe17-6854-4ef1-ae47-288f2dbafb60-tls-certs\") pod \"test-model-56d1d9-kserve-65d4df9bdf-7n97s\" (UID: \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\") " pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" Oct 30 01:13:13.885368 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.885343 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-tmp-dir\") pod \"test-model-56d1d9-kserve-65d4df9bdf-7n97s\" (UID: \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\") " pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" Oct 30 01:13:13.885789 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.885751 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-dshm\") pod \"test-model-56d1d9-kserve-65d4df9bdf-7n97s\" (UID: \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\") " pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" Oct 30 01:13:13.886045 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.886025 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-home\") pod \"test-model-56d1d9-kserve-65d4df9bdf-7n97s\" (UID: \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\") " pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" Oct 30 01:13:13.886253 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.886238 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq54m\" (UniqueName: \"kubernetes.io/projected/e84bfe17-6854-4ef1-ae47-288f2dbafb60-kube-api-access-zq54m\") pod \"test-model-56d1d9-kserve-65d4df9bdf-7n97s\" (UID: \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\") " pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" Oct 30 01:13:13.988588 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.987698 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-model-cache\") pod \"test-model-56d1d9-kserve-65d4df9bdf-7n97s\" (UID: \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\") " pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" Oct 30 01:13:13.988588 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.987783 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e84bfe17-6854-4ef1-ae47-288f2dbafb60-tls-certs\") pod \"test-model-56d1d9-kserve-65d4df9bdf-7n97s\" (UID: \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\") " pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" Oct 30 01:13:13.988588 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.987963 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-tmp-dir\") pod \"test-model-56d1d9-kserve-65d4df9bdf-7n97s\" (UID: \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\") " pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" Oct 30 01:13:13.988588 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.988095 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-dshm\") pod \"test-model-56d1d9-kserve-65d4df9bdf-7n97s\" (UID: \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\") " pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" Oct 30 01:13:13.988588 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.988193 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-home\") pod \"test-model-56d1d9-kserve-65d4df9bdf-7n97s\" (UID: \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\") " pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" Oct 30 01:13:13.988588 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.988275 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq54m\" (UniqueName: \"kubernetes.io/projected/e84bfe17-6854-4ef1-ae47-288f2dbafb60-kube-api-access-zq54m\") pod \"test-model-56d1d9-kserve-65d4df9bdf-7n97s\" (UID: \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\") " pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" Oct 30 01:13:13.990030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.989806 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-tmp-dir\") pod \"test-model-56d1d9-kserve-65d4df9bdf-7n97s\" (UID: \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\") " pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" Oct 30 01:13:13.990175 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.990132 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-home\") pod \"test-model-56d1d9-kserve-65d4df9bdf-7n97s\" (UID: \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\") " pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" Oct 30 01:13:13.990264 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.990194 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-model-cache\") pod \"test-model-56d1d9-kserve-65d4df9bdf-7n97s\" (UID: \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\") " pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" Oct 30 01:13:13.992817 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.992747 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-dshm\") pod \"test-model-56d1d9-kserve-65d4df9bdf-7n97s\" (UID: \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\") " pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" Oct 30 01:13:13.998819 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:13.998278 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e84bfe17-6854-4ef1-ae47-288f2dbafb60-tls-certs\") pod \"test-model-56d1d9-kserve-65d4df9bdf-7n97s\" (UID: \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\") " pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" Oct 30 01:13:14.012132 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:14.012092 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq54m\" (UniqueName: \"kubernetes.io/projected/e84bfe17-6854-4ef1-ae47-288f2dbafb60-kube-api-access-zq54m\") pod \"test-model-56d1d9-kserve-65d4df9bdf-7n97s\" (UID: \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\") " pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" Oct 30 01:13:14.049804 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:14.038535 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:14.062379 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:14.062319 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" Oct 30 01:13:14.411495 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:13:14.411465 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode84bfe17_6854_4ef1_ae47_288f2dbafb60.slice/crio-92be1bc8de66e976b68a973a0a749bd8b63cfb96d5d4381d6683b9d8b1303cfe WatchSource:0}: Error finding container 92be1bc8de66e976b68a973a0a749bd8b63cfb96d5d4381d6683b9d8b1303cfe: Status 404 returned error can't find the container with id 92be1bc8de66e976b68a973a0a749bd8b63cfb96d5d4381d6683b9d8b1303cfe Oct 30 01:13:14.412229 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:14.412168 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s"] Oct 30 01:13:14.614257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:14.614218 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" event={"ID":"e84bfe17-6854-4ef1-ae47-288f2dbafb60","Type":"ContainerStarted","Data":"2ca56166e386ed2f993690735de91adadd9c1df12a813a57efd3b8e8ae42caa3"} Oct 30 01:13:14.614517 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:14.614492 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" event={"ID":"e84bfe17-6854-4ef1-ae47-288f2dbafb60","Type":"ContainerStarted","Data":"92be1bc8de66e976b68a973a0a749bd8b63cfb96d5d4381d6683b9d8b1303cfe"} Oct 30 01:13:14.667573 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:14.667469 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" podStartSLOduration=1.667454166 podStartE2EDuration="1.667454166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:14.663891025 +0000 UTC m=+1794.979759350" watchObservedRunningTime="2026-10-30 01:13:14.667454166 +0000 UTC m=+1794.983322491" Oct 30 01:13:15.063102 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:15.063062 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" Oct 30 01:13:15.597368 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:15.597331 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:15.700216 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:15.700159 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:15.793619 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:15.793588 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:15.913943 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:15.913650 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:17.571496 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:17.571449 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:19.351054 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:19.350396 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:19.671844 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:19.671694 5300 generic.go:356] "Generic (PLEG): container finished" podID="19cf7721-345d-4937-b699-292dcdde5ef0" containerID="e80351d009fc147f9cf2d48c73e241c831642b61befc5cd9629fd51da54e82bc" exitCode=0 Oct 30 01:13:19.671844 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:19.671778 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" event={"ID":"19cf7721-345d-4937-b699-292dcdde5ef0","Type":"ContainerDied","Data":"e80351d009fc147f9cf2d48c73e241c831642b61befc5cd9629fd51da54e82bc"} Oct 30 01:13:19.672297 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:19.672278 5300 scope.go:122] "RemoveContainer" containerID="e80351d009fc147f9cf2d48c73e241c831642b61befc5cd9629fd51da54e82bc" Oct 30 01:13:19.674073 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:19.674040 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:13:19.674166 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:19.674113 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:13:19.674166 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:19.674147 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:13:20.681005 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:20.680965 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" event={"ID":"19cf7721-345d-4937-b699-292dcdde5ef0","Type":"ContainerStarted","Data":"4b07d4946a36dc0b5659d2c75e8d20ab12b204836953b8a8f44c3e1184203aa4"} Oct 30 01:13:20.911305 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:20.911267 5300 scope.go:122] "RemoveContainer" containerID="017ed38f55ce4b5a817d8e5863c39dafb727cc66d33f24b33d6bb300359b470b" Oct 30 01:13:21.087823 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:21.087750 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:21.542986 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:21.542946 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:13:21.543351 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:21.543322 5300 patch_prober.go:29] interesting pod/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 01:13:21.543460 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:21.543385 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" podUID="19cf7721-345d-4937-b699-292dcdde5ef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 01:13:21.596637 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:13:21.596596 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:13:22.545910 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:22.544158 5300 patch_prober.go:29] interesting pod/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 01:13:22.545910 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:22.544222 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" podUID="19cf7721-345d-4937-b699-292dcdde5ef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 01:13:22.597286 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:22.597250 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:22.863059 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:22.851546 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:22.985346 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:22.985290 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:23.542208 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:23.542173 5300 patch_prober.go:29] interesting pod/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 01:13:23.542519 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:23.542474 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" podUID="19cf7721-345d-4937-b699-292dcdde5ef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 01:13:23.548291 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:23.548254 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:24.063082 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:24.063041 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" Oct 30 01:13:24.082437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:24.082401 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" Oct 30 01:13:24.548257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:24.548218 5300 patch_prober.go:29] interesting pod/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 01:13:24.548631 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:24.548546 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" podUID="19cf7721-345d-4937-b699-292dcdde5ef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 01:13:24.804456 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:24.802917 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" Oct 30 01:13:24.986072 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:24.986025 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:25.101544 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:25.101460 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:25.543279 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:25.543239 5300 patch_prober.go:29] interesting pod/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 30 01:13:25.543479 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:25.543325 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" podUID="19cf7721-345d-4937-b699-292dcdde5ef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 30 01:13:26.547430 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:26.547396 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:13:26.547897 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:26.547875 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:13:26.548773 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:26.548746 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:13:26.743031 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:26.742998 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:28.353705 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:28.353645 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:30.317270 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:30.317226 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:31.217570 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:31.217481 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:31.317966 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:31.317920 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:31.634877 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:31.634840 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:32.067107 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:32.067068 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:32.166400 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:32.166366 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:32.298786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:32.293702 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:33.029975 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:33.029932 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:33.904138 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:33.904098 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:35.385094 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:35.385055 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:35.486046 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:35.485995 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:36.053272 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:36.053233 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:36.216141 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:36.216090 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:37.794070 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:37.794017 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:38.083855 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:38.083719 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:38.598080 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:38.598020 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:38.701228 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:38.701175 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:40.047553 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:40.047519 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:41.794595 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:41.794562 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:43.474032 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:43.473995 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:45.170529 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:45.170482 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:46.825210 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:46.825147 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:48.619163 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:48.619124 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:50.339479 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:50.339431 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:51.600959 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:13:51.600924 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:13:51.941792 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:51.941673 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:52.160268 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:52.160232 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:53.839651 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:53.839616 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:55.639472 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:55.639430 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:57.279727 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:57.279676 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:13:59.195373 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:13:59.195337 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:00.750150 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:00.750103 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:02.471368 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:02.471330 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:04.358716 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:04.358677 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:05.943125 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:05.943089 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:06.040296 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:06.040255 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:07.639603 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:07.639565 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:09.375347 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:09.375309 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:11.155503 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:11.155465 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:11.481033 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:11.480992 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:11.582943 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:11.582908 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:11.839464 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:11.839359 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:11.947564 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:11.947527 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:12.940159 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:12.940125 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:14.523636 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:14.523594 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:16.249536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:16.249504 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:18.118328 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:18.118275 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:20.217894 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:20.217844 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:20.371302 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:20.371245 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:21.104139 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:21.104101 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:21.596714 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:14:21.596683 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:14:22.351034 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:22.350995 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:24.054126 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:24.054081 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:24.167691 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:24.167652 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:25.993454 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:25.993411 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:27.683052 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:27.683013 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:29.437425 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:29.437333 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:29.511260 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:29.511223 5300 generic.go:356] "Generic (PLEG): container finished" podID="c908ef75-1950-4c44-8311-47bcfd6b814a" containerID="100d055f2312e10c435768d93aefdefa6a92e5c3e5682a40b472c1cda81a0e1d" exitCode=0 Oct 30 01:14:29.511540 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:29.511304 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" event={"ID":"c908ef75-1950-4c44-8311-47bcfd6b814a","Type":"ContainerDied","Data":"100d055f2312e10c435768d93aefdefa6a92e5c3e5682a40b472c1cda81a0e1d"} Oct 30 01:14:29.511998 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:29.511961 5300 scope.go:122] "RemoveContainer" containerID="100d055f2312e10c435768d93aefdefa6a92e5c3e5682a40b472c1cda81a0e1d" Oct 30 01:14:29.514352 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:29.514316 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:14:29.514442 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:29.514395 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:14:29.514442 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:29.514431 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:14:30.555389 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:30.555351 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" event={"ID":"c908ef75-1950-4c44-8311-47bcfd6b814a","Type":"ContainerStarted","Data":"eecc51af3bc02276f9b01844f0546ed19ca06bcb660ce94432e2bfcd2361bc3c"} Oct 30 01:14:31.192947 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:31.192908 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:31.312622 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:31.312576 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:14:31.314636 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:31.314604 5300 patch_prober.go:29] interesting pod/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 01:14:31.314839 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:31.314661 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" podUID="c908ef75-1950-4c44-8311-47bcfd6b814a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 01:14:32.312848 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:32.312809 5300 patch_prober.go:29] interesting pod/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 01:14:32.313274 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:32.312874 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" podUID="c908ef75-1950-4c44-8311-47bcfd6b814a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 01:14:32.592148 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:32.592062 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:33.102496 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:33.102463 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:33.204715 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:33.204677 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:33.313250 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:33.313214 5300 patch_prober.go:29] interesting pod/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 01:14:33.313644 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:33.313270 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" podUID="c908ef75-1950-4c44-8311-47bcfd6b814a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 01:14:34.312528 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:34.312495 5300 patch_prober.go:29] interesting pod/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 01:14:34.312717 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:34.312556 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" podUID="c908ef75-1950-4c44-8311-47bcfd6b814a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 01:14:35.077602 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:35.077563 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:35.314715 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:35.313453 5300 patch_prober.go:29] interesting pod/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 30 01:14:35.314715 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:35.313516 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" podUID="c908ef75-1950-4c44-8311-47bcfd6b814a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 30 01:14:36.318772 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:36.318731 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:14:36.319234 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:36.319166 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:14:36.319848 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:36.319825 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:14:36.739226 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:36.739194 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:38.434470 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:38.434434 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:40.257243 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:40.257191 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:41.900374 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:41.900338 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:43.507313 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:43.507277 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:43.931199 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:43.931098 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:44.034726 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:44.034689 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:44.787310 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:44.787263 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:45.545570 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:45.545533 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:47.274782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:47.274504 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:49.251352 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:49.251315 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:51.051689 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:51.051638 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:51.597354 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:14:51.597314 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:14:52.840337 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:52.840287 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:54.420542 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:54.420506 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:56.160967 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:56.160931 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:57.831553 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:57.831512 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:14:59.427207 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:14:59.427169 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:00.295060 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.295024 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld"] Oct 30 01:15:00.352421 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.352381 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh"] Oct 30 01:15:00.357167 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.357132 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld" Oct 30 01:15:00.400196 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.389214 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld"] Oct 30 01:15:00.400196 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.389263 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj"] Oct 30 01:15:00.400196 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.393299 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh" Oct 30 01:15:00.400196 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.394993 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888715-b729h"] Oct 30 01:15:00.400196 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.396517 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj" Oct 30 01:15:00.400509 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.400346 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh"] Oct 30 01:15:00.400509 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.400432 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-b729h" Oct 30 01:15:00.406554 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.406528 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv"] Oct 30 01:15:00.414933 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.414899 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t"] Oct 30 01:15:00.416230 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.416210 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv" Oct 30 01:15:00.422273 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.422237 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888715-b729h"] Oct 30 01:15:00.422424 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.422348 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t" Oct 30 01:15:00.424437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.424414 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv"] Oct 30 01:15:00.430285 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.430262 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t"] Oct 30 01:15:00.436792 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.436739 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj"] Oct 30 01:15:00.460776 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.460732 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5abe0803-281e-425c-8f7d-1c99c84c264f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld\" (UID: \"5abe0803-281e-425c-8f7d-1c99c84c264f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld" Oct 30 01:15:00.461106 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.461084 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v59j2\" (UniqueName: \"kubernetes.io/projected/7057bdfa-7b6b-4183-bda8-345f355c8567-kube-api-access-v59j2\") pod \"maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj\" (UID: \"7057bdfa-7b6b-4183-bda8-345f355c8567\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj" Oct 30 01:15:00.461292 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.461277 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eacc79bd-911f-4694-a1ae-deaca2d2df02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh\" (UID: \"eacc79bd-911f-4694-a1ae-deaca2d2df02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh" Oct 30 01:15:00.461448 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.461434 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7057bdfa-7b6b-4183-bda8-345f355c8567-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj\" (UID: \"7057bdfa-7b6b-4183-bda8-345f355c8567\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj" Oct 30 01:15:00.461651 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.461635 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkdbz\" (UniqueName: \"kubernetes.io/projected/eacc79bd-911f-4694-a1ae-deaca2d2df02-kube-api-access-tkdbz\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh\" (UID: \"eacc79bd-911f-4694-a1ae-deaca2d2df02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh" Oct 30 01:15:00.461914 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.461897 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46xs5\" (UniqueName: \"kubernetes.io/projected/5abe0803-281e-425c-8f7d-1c99c84c264f-kube-api-access-46xs5\") pod \"maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld\" (UID: \"5abe0803-281e-425c-8f7d-1c99c84c264f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld" Oct 30 01:15:00.466292 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.466271 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\"" Oct 30 01:15:00.564153 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.564047 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70bc1eab-1e25-4afb-b6c2-6697d1a77c21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t\" (UID: \"70bc1eab-1e25-4afb-b6c2-6697d1a77c21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t" Oct 30 01:15:00.564153 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.564112 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v59j2\" (UniqueName: \"kubernetes.io/projected/7057bdfa-7b6b-4183-bda8-345f355c8567-kube-api-access-v59j2\") pod \"maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj\" (UID: \"7057bdfa-7b6b-4183-bda8-345f355c8567\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj" Oct 30 01:15:00.564366 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.564199 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4hsw\" (UniqueName: \"kubernetes.io/projected/ee041d48-724e-47ae-a69e-2906033b9ce4-kube-api-access-s4hsw\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv\" (UID: \"ee041d48-724e-47ae-a69e-2906033b9ce4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv" Oct 30 01:15:00.564366 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.564254 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dab0f86-2737-433f-a285-c2719d960bcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888715-b729h\" (UID: \"0dab0f86-2737-433f-a285-c2719d960bcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-b729h" Oct 30 01:15:00.564366 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.564331 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eacc79bd-911f-4694-a1ae-deaca2d2df02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh\" (UID: \"eacc79bd-911f-4694-a1ae-deaca2d2df02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh" Oct 30 01:15:00.564489 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.564364 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq5n4\" (UniqueName: \"kubernetes.io/projected/0dab0f86-2737-433f-a285-c2719d960bcd-kube-api-access-pq5n4\") pod \"maas-api-key-cleanup-29888715-b729h\" (UID: \"0dab0f86-2737-433f-a285-c2719d960bcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-b729h" Oct 30 01:15:00.564489 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.564436 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7057bdfa-7b6b-4183-bda8-345f355c8567-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj\" (UID: \"7057bdfa-7b6b-4183-bda8-345f355c8567\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj" Oct 30 01:15:00.564574 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.564485 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee041d48-724e-47ae-a69e-2906033b9ce4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv\" (UID: \"ee041d48-724e-47ae-a69e-2906033b9ce4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv" Oct 30 01:15:00.564631 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.564602 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkdbz\" (UniqueName: \"kubernetes.io/projected/eacc79bd-911f-4694-a1ae-deaca2d2df02-kube-api-access-tkdbz\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh\" (UID: \"eacc79bd-911f-4694-a1ae-deaca2d2df02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh" Oct 30 01:15:00.564821 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.564798 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46xs5\" (UniqueName: \"kubernetes.io/projected/5abe0803-281e-425c-8f7d-1c99c84c264f-kube-api-access-46xs5\") pod \"maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld\" (UID: \"5abe0803-281e-425c-8f7d-1c99c84c264f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld" Oct 30 01:15:00.564966 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.564835 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whvp6\" (UniqueName: \"kubernetes.io/projected/70bc1eab-1e25-4afb-b6c2-6697d1a77c21-kube-api-access-whvp6\") pod \"maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t\" (UID: \"70bc1eab-1e25-4afb-b6c2-6697d1a77c21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t" Oct 30 01:15:00.564966 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.564926 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5abe0803-281e-425c-8f7d-1c99c84c264f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld\" (UID: \"5abe0803-281e-425c-8f7d-1c99c84c264f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld" Oct 30 01:15:00.565140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.565122 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eacc79bd-911f-4694-a1ae-deaca2d2df02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh\" (UID: \"eacc79bd-911f-4694-a1ae-deaca2d2df02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh" Oct 30 01:15:00.565192 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.565168 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7057bdfa-7b6b-4183-bda8-345f355c8567-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj\" (UID: \"7057bdfa-7b6b-4183-bda8-345f355c8567\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj" Oct 30 01:15:00.565563 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.565535 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5abe0803-281e-425c-8f7d-1c99c84c264f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld\" (UID: \"5abe0803-281e-425c-8f7d-1c99c84c264f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld" Oct 30 01:15:00.594598 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.594568 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v59j2\" (UniqueName: \"kubernetes.io/projected/7057bdfa-7b6b-4183-bda8-345f355c8567-kube-api-access-v59j2\") pod \"maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj\" (UID: \"7057bdfa-7b6b-4183-bda8-345f355c8567\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj" Oct 30 01:15:00.594598 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.594590 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46xs5\" (UniqueName: \"kubernetes.io/projected/5abe0803-281e-425c-8f7d-1c99c84c264f-kube-api-access-46xs5\") pod \"maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld\" (UID: \"5abe0803-281e-425c-8f7d-1c99c84c264f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld" Oct 30 01:15:00.595117 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.595085 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:00.597909 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.597886 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkdbz\" (UniqueName: \"kubernetes.io/projected/eacc79bd-911f-4694-a1ae-deaca2d2df02-kube-api-access-tkdbz\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh\" (UID: \"eacc79bd-911f-4694-a1ae-deaca2d2df02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh" Oct 30 01:15:00.665896 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.665857 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq5n4\" (UniqueName: \"kubernetes.io/projected/0dab0f86-2737-433f-a285-c2719d960bcd-kube-api-access-pq5n4\") pod \"maas-api-key-cleanup-29888715-b729h\" (UID: \"0dab0f86-2737-433f-a285-c2719d960bcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-b729h" Oct 30 01:15:00.666109 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.665941 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee041d48-724e-47ae-a69e-2906033b9ce4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv\" (UID: \"ee041d48-724e-47ae-a69e-2906033b9ce4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv" Oct 30 01:15:00.666109 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.666093 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whvp6\" (UniqueName: \"kubernetes.io/projected/70bc1eab-1e25-4afb-b6c2-6697d1a77c21-kube-api-access-whvp6\") pod \"maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t\" (UID: \"70bc1eab-1e25-4afb-b6c2-6697d1a77c21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t" Oct 30 01:15:00.666224 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.666184 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70bc1eab-1e25-4afb-b6c2-6697d1a77c21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t\" (UID: \"70bc1eab-1e25-4afb-b6c2-6697d1a77c21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t" Oct 30 01:15:00.666285 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.666232 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4hsw\" (UniqueName: \"kubernetes.io/projected/ee041d48-724e-47ae-a69e-2906033b9ce4-kube-api-access-s4hsw\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv\" (UID: \"ee041d48-724e-47ae-a69e-2906033b9ce4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv" Oct 30 01:15:00.666285 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.666264 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dab0f86-2737-433f-a285-c2719d960bcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888715-b729h\" (UID: \"0dab0f86-2737-433f-a285-c2719d960bcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-b729h" Oct 30 01:15:00.666713 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.666688 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee041d48-724e-47ae-a69e-2906033b9ce4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv\" (UID: \"ee041d48-724e-47ae-a69e-2906033b9ce4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv" Oct 30 01:15:00.666915 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.666893 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dab0f86-2737-433f-a285-c2719d960bcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888715-b729h\" (UID: \"0dab0f86-2737-433f-a285-c2719d960bcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-b729h" Oct 30 01:15:00.666988 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.666910 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70bc1eab-1e25-4afb-b6c2-6697d1a77c21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t\" (UID: \"70bc1eab-1e25-4afb-b6c2-6697d1a77c21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t" Oct 30 01:15:00.692116 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.692089 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whvp6\" (UniqueName: \"kubernetes.io/projected/70bc1eab-1e25-4afb-b6c2-6697d1a77c21-kube-api-access-whvp6\") pod \"maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t\" (UID: \"70bc1eab-1e25-4afb-b6c2-6697d1a77c21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t" Oct 30 01:15:00.694778 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.694734 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4hsw\" (UniqueName: \"kubernetes.io/projected/ee041d48-724e-47ae-a69e-2906033b9ce4-kube-api-access-s4hsw\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv\" (UID: \"ee041d48-724e-47ae-a69e-2906033b9ce4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv" Oct 30 01:15:00.694952 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.694935 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq5n4\" (UniqueName: \"kubernetes.io/projected/0dab0f86-2737-433f-a285-c2719d960bcd-kube-api-access-pq5n4\") pod \"maas-api-key-cleanup-29888715-b729h\" (UID: \"0dab0f86-2737-433f-a285-c2719d960bcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-b729h" Oct 30 01:15:00.719060 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.719024 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld" Oct 30 01:15:00.720934 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.720901 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:00.734865 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.734838 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh" Oct 30 01:15:00.749382 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.749345 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj" Oct 30 01:15:00.765090 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.765052 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-b729h" Oct 30 01:15:00.775376 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.775348 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv" Oct 30 01:15:00.783173 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:00.783125 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t" Oct 30 01:15:01.206559 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:01.204573 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t"] Oct 30 01:15:01.218198 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:15:01.214394 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70bc1eab_1e25_4afb_b6c2_6697d1a77c21.slice/crio-48b1d802c944408a226487f9d001bd88843449635be026f881c992288570d020 WatchSource:0}: Error finding container 48b1d802c944408a226487f9d001bd88843449635be026f881c992288570d020: Status 404 returned error can't find the container with id 48b1d802c944408a226487f9d001bd88843449635be026f881c992288570d020 Oct 30 01:15:01.220790 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:01.220753 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv"] Oct 30 01:15:01.223285 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:15:01.223261 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee041d48_724e_47ae_a69e_2906033b9ce4.slice/crio-f7f880bc391baf659330fe7ab8b3f55b72d61aebc46e7be879ae0766bd69ceed WatchSource:0}: Error finding container f7f880bc391baf659330fe7ab8b3f55b72d61aebc46e7be879ae0766bd69ceed: Status 404 returned error can't find the container with id f7f880bc391baf659330fe7ab8b3f55b72d61aebc46e7be879ae0766bd69ceed Oct 30 01:15:01.245609 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:01.245580 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld"] Oct 30 01:15:01.250979 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:15:01.250948 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abe0803_281e_425c_8f7d_1c99c84c264f.slice/crio-0fff320956427be4177bbd599018df34f6106852513cc82136f41e36dc5f1acc WatchSource:0}: Error finding container 0fff320956427be4177bbd599018df34f6106852513cc82136f41e36dc5f1acc: Status 404 returned error can't find the container with id 0fff320956427be4177bbd599018df34f6106852513cc82136f41e36dc5f1acc Oct 30 01:15:01.279343 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:15:01.279313 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeacc79bd_911f_4694_a1ae_deaca2d2df02.slice/crio-012c7a5b3b727c98e98273e83d1e57e4c2237a7ee31e6b2f9849f8368ebd9eb3 WatchSource:0}: Error finding container 012c7a5b3b727c98e98273e83d1e57e4c2237a7ee31e6b2f9849f8368ebd9eb3: Status 404 returned error can't find the container with id 012c7a5b3b727c98e98273e83d1e57e4c2237a7ee31e6b2f9849f8368ebd9eb3 Oct 30 01:15:01.279739 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:01.279664 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh"] Oct 30 01:15:01.294065 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:15:01.294021 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057bdfa_7b6b_4183_bda8_345f355c8567.slice/crio-6a8abb3b6776e2c2476988ab93d1af8daa736785cf487572b7add4e13d1e84cd WatchSource:0}: Error finding container 6a8abb3b6776e2c2476988ab93d1af8daa736785cf487572b7add4e13d1e84cd: Status 404 returned error can't find the container with id 6a8abb3b6776e2c2476988ab93d1af8daa736785cf487572b7add4e13d1e84cd Oct 30 01:15:01.294203 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:01.294146 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:01.294474 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:01.294452 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj"] Oct 30 01:15:01.375930 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:01.375904 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888715-b729h"] Oct 30 01:15:01.524476 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:15:01.524352 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dab0f86_2737_433f_a285_c2719d960bcd.slice/crio-c50433629b037638104424dab684e4ea79e001417c697d7dd8dac6c38cbf6324 WatchSource:0}: Error finding container c50433629b037638104424dab684e4ea79e001417c697d7dd8dac6c38cbf6324: Status 404 returned error can't find the container with id c50433629b037638104424dab684e4ea79e001417c697d7dd8dac6c38cbf6324 Oct 30 01:15:01.857987 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:01.857949 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t" event={"ID":"70bc1eab-1e25-4afb-b6c2-6697d1a77c21","Type":"ContainerStarted","Data":"f922ccb20e1331d489dc317e9c8dc4ba309b1e2d617f7aa8c5d4bc6e6decaca4"} Oct 30 01:15:01.858161 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:01.858014 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t" event={"ID":"70bc1eab-1e25-4afb-b6c2-6697d1a77c21","Type":"ContainerStarted","Data":"48b1d802c944408a226487f9d001bd88843449635be026f881c992288570d020"} Oct 30 01:15:01.859599 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:01.859573 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-b729h" event={"ID":"0dab0f86-2737-433f-a285-c2719d960bcd","Type":"ContainerStarted","Data":"c50433629b037638104424dab684e4ea79e001417c697d7dd8dac6c38cbf6324"} Oct 30 01:15:01.861253 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:01.861223 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh" event={"ID":"eacc79bd-911f-4694-a1ae-deaca2d2df02","Type":"ContainerStarted","Data":"a7a82bb3a6ebbf4aff00db45843b745ba26150076d7edb40c49681d1dae059b6"} Oct 30 01:15:01.861384 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:01.861280 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh" event={"ID":"eacc79bd-911f-4694-a1ae-deaca2d2df02","Type":"ContainerStarted","Data":"012c7a5b3b727c98e98273e83d1e57e4c2237a7ee31e6b2f9849f8368ebd9eb3"} Oct 30 01:15:01.863081 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:01.863058 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv" event={"ID":"ee041d48-724e-47ae-a69e-2906033b9ce4","Type":"ContainerStarted","Data":"81633c8e618f9bac8ce3fe143616c485fea76912f91199029e6849728d94c158"} Oct 30 01:15:01.863196 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:01.863100 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv" event={"ID":"ee041d48-724e-47ae-a69e-2906033b9ce4","Type":"ContainerStarted","Data":"f7f880bc391baf659330fe7ab8b3f55b72d61aebc46e7be879ae0766bd69ceed"} Oct 30 01:15:01.864817 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:01.864793 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld" event={"ID":"5abe0803-281e-425c-8f7d-1c99c84c264f","Type":"ContainerStarted","Data":"91e6bba5a35da1f7a56f04bc4a29a2f6c188a48f203233761fbc82da002dba0b"} Oct 30 01:15:01.864917 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:01.864839 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld" event={"ID":"5abe0803-281e-425c-8f7d-1c99c84c264f","Type":"ContainerStarted","Data":"0fff320956427be4177bbd599018df34f6106852513cc82136f41e36dc5f1acc"} Oct 30 01:15:01.866999 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:01.866978 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj" event={"ID":"7057bdfa-7b6b-4183-bda8-345f355c8567","Type":"ContainerStarted","Data":"e590d597c04a5d4baf67082462b286cbec1f71f810cb6825ffa47bef443a7d4d"} Oct 30 01:15:01.867112 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:01.867011 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj" event={"ID":"7057bdfa-7b6b-4183-bda8-345f355c8567","Type":"ContainerStarted","Data":"6a8abb3b6776e2c2476988ab93d1af8daa736785cf487572b7add4e13d1e84cd"} Oct 30 01:15:01.902913 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:01.902856 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t" podStartSLOduration=1.9028422379999999 podStartE2EDuration="1.902842238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:01.899529351 +0000 UTC m=+1902.215397676" watchObservedRunningTime="2026-10-30 01:15:01.902842238 +0000 UTC m=+1902.218710562" Oct 30 01:15:01.943841 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:01.943791 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv" podStartSLOduration=1.943776884 podStartE2EDuration="1.943776884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:01.939778421 +0000 UTC m=+1902.255646746" watchObservedRunningTime="2026-10-30 01:15:01.943776884 +0000 UTC m=+1902.259645206" Oct 30 01:15:01.985146 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:01.985089 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld" podStartSLOduration=1.985074022 podStartE2EDuration="1.985074022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:01.982632401 +0000 UTC m=+1902.298500738" watchObservedRunningTime="2026-10-30 01:15:01.985074022 +0000 UTC m=+1902.300942348" Oct 30 01:15:02.025179 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:02.025120 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh" podStartSLOduration=2.025104359 podStartE2EDuration="2.025104359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:02.023575053 +0000 UTC m=+1902.339443378" watchObservedRunningTime="2026-10-30 01:15:02.025104359 +0000 UTC m=+1902.340972684" Oct 30 01:15:02.067953 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:02.067892 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj" podStartSLOduration=2.067875935 podStartE2EDuration="2.067875935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:02.063440785 +0000 UTC m=+1902.379309110" watchObservedRunningTime="2026-10-30 01:15:02.067875935 +0000 UTC m=+1902.383744305" Oct 30 01:15:02.876845 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:02.876803 5300 generic.go:356] "Generic (PLEG): container finished" podID="70bc1eab-1e25-4afb-b6c2-6697d1a77c21" containerID="f922ccb20e1331d489dc317e9c8dc4ba309b1e2d617f7aa8c5d4bc6e6decaca4" exitCode=60 Oct 30 01:15:02.877373 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:02.876889 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t" event={"ID":"70bc1eab-1e25-4afb-b6c2-6697d1a77c21","Type":"ContainerDied","Data":"f922ccb20e1331d489dc317e9c8dc4ba309b1e2d617f7aa8c5d4bc6e6decaca4"} Oct 30 01:15:02.877373 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:02.877263 5300 scope.go:122] "RemoveContainer" containerID="f922ccb20e1331d489dc317e9c8dc4ba309b1e2d617f7aa8c5d4bc6e6decaca4" Oct 30 01:15:02.878660 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:02.878640 5300 generic.go:356] "Generic (PLEG): container finished" podID="0dab0f86-2737-433f-a285-c2719d960bcd" containerID="7e6d8e2f7b521711a27087c490c8bb005a9f342b663cda3e96f645836a02d0f6" exitCode=60 Oct 30 01:15:02.878808 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:02.878719 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-b729h" event={"ID":"0dab0f86-2737-433f-a285-c2719d960bcd","Type":"ContainerDied","Data":"7e6d8e2f7b521711a27087c490c8bb005a9f342b663cda3e96f645836a02d0f6"} Oct 30 01:15:02.879022 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:02.879004 5300 scope.go:122] "RemoveContainer" containerID="7e6d8e2f7b521711a27087c490c8bb005a9f342b663cda3e96f645836a02d0f6" Oct 30 01:15:02.880736 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:02.880664 5300 generic.go:356] "Generic (PLEG): container finished" podID="eacc79bd-911f-4694-a1ae-deaca2d2df02" containerID="a7a82bb3a6ebbf4aff00db45843b745ba26150076d7edb40c49681d1dae059b6" exitCode=60 Oct 30 01:15:02.880847 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:02.880735 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh" event={"ID":"eacc79bd-911f-4694-a1ae-deaca2d2df02","Type":"ContainerDied","Data":"a7a82bb3a6ebbf4aff00db45843b745ba26150076d7edb40c49681d1dae059b6"} Oct 30 01:15:02.881076 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:02.881060 5300 scope.go:122] "RemoveContainer" containerID="a7a82bb3a6ebbf4aff00db45843b745ba26150076d7edb40c49681d1dae059b6" Oct 30 01:15:02.882993 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:02.882970 5300 generic.go:356] "Generic (PLEG): container finished" podID="ee041d48-724e-47ae-a69e-2906033b9ce4" containerID="81633c8e618f9bac8ce3fe143616c485fea76912f91199029e6849728d94c158" exitCode=60 Oct 30 01:15:02.883109 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:02.883051 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv" event={"ID":"ee041d48-724e-47ae-a69e-2906033b9ce4","Type":"ContainerDied","Data":"81633c8e618f9bac8ce3fe143616c485fea76912f91199029e6849728d94c158"} Oct 30 01:15:02.883320 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:02.883300 5300 scope.go:122] "RemoveContainer" containerID="81633c8e618f9bac8ce3fe143616c485fea76912f91199029e6849728d94c158" Oct 30 01:15:02.884598 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:02.884578 5300 generic.go:356] "Generic (PLEG): container finished" podID="5abe0803-281e-425c-8f7d-1c99c84c264f" containerID="91e6bba5a35da1f7a56f04bc4a29a2f6c188a48f203233761fbc82da002dba0b" exitCode=60 Oct 30 01:15:02.884704 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:02.884653 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld" event={"ID":"5abe0803-281e-425c-8f7d-1c99c84c264f","Type":"ContainerDied","Data":"91e6bba5a35da1f7a56f04bc4a29a2f6c188a48f203233761fbc82da002dba0b"} Oct 30 01:15:02.884868 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:02.884851 5300 scope.go:122] "RemoveContainer" containerID="91e6bba5a35da1f7a56f04bc4a29a2f6c188a48f203233761fbc82da002dba0b" Oct 30 01:15:02.886275 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:02.886256 5300 generic.go:356] "Generic (PLEG): container finished" podID="7057bdfa-7b6b-4183-bda8-345f355c8567" containerID="e590d597c04a5d4baf67082462b286cbec1f71f810cb6825ffa47bef443a7d4d" exitCode=60 Oct 30 01:15:02.886356 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:02.886281 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj" event={"ID":"7057bdfa-7b6b-4183-bda8-345f355c8567","Type":"ContainerDied","Data":"e590d597c04a5d4baf67082462b286cbec1f71f810cb6825ffa47bef443a7d4d"} Oct 30 01:15:02.886532 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:02.886518 5300 scope.go:122] "RemoveContainer" containerID="e590d597c04a5d4baf67082462b286cbec1f71f810cb6825ffa47bef443a7d4d" Oct 30 01:15:03.011008 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:03.010960 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:03.896187 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:03.896091 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh" event={"ID":"eacc79bd-911f-4694-a1ae-deaca2d2df02","Type":"ContainerStarted","Data":"c66202252d776d341d31f369046b9c3db0946a4e3bc8f13e5bf85ef25fb51aa9"} Oct 30 01:15:03.898197 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:03.898173 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv" event={"ID":"ee041d48-724e-47ae-a69e-2906033b9ce4","Type":"ContainerStarted","Data":"f370b17fb04b2f822f4d41e47180a50369d2971ce017f295d5b6fa7e7f6613ae"} Oct 30 01:15:03.900110 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:03.900086 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld" event={"ID":"5abe0803-281e-425c-8f7d-1c99c84c264f","Type":"ContainerStarted","Data":"725140932cffe7c649c8dc2e864be35a46ae2c9745a87dc802e832ae8e33e16c"} Oct 30 01:15:03.902085 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:03.902058 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj" event={"ID":"7057bdfa-7b6b-4183-bda8-345f355c8567","Type":"ContainerStarted","Data":"c89c970f499bb369d8fc474918768883b88336f78469ed98fbb3f737ec2656d6"} Oct 30 01:15:03.904162 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:03.904138 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t" event={"ID":"70bc1eab-1e25-4afb-b6c2-6697d1a77c21","Type":"ContainerStarted","Data":"e3a6531d67b163b4882142a56e55f3d35ae42d53f5e6482df2bab0b86cc7a998"} Oct 30 01:15:03.906293 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:03.906272 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-b729h" event={"ID":"0dab0f86-2737-433f-a285-c2719d960bcd","Type":"ContainerStarted","Data":"2cf55a0c54ddd792a13be31379208a66e6e27c8db07b1ad2885afe84ebd15267"} Oct 30 01:15:04.067714 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:04.067657 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-b729h" podStartSLOduration=4.067642975 podStartE2EDuration="4.067642975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:04.064422087 +0000 UTC m=+1904.380290414" watchObservedRunningTime="2026-10-30 01:15:04.067642975 +0000 UTC m=+1904.383511339" Oct 30 01:15:04.763098 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:04.763063 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:04.918174 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:04.918136 5300 generic.go:356] "Generic (PLEG): container finished" podID="eacc79bd-911f-4694-a1ae-deaca2d2df02" containerID="c66202252d776d341d31f369046b9c3db0946a4e3bc8f13e5bf85ef25fb51aa9" exitCode=60 Oct 30 01:15:04.918661 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:04.918227 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh" event={"ID":"eacc79bd-911f-4694-a1ae-deaca2d2df02","Type":"ContainerDied","Data":"c66202252d776d341d31f369046b9c3db0946a4e3bc8f13e5bf85ef25fb51aa9"} Oct 30 01:15:04.918661 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:04.918283 5300 scope.go:122] "RemoveContainer" containerID="a7a82bb3a6ebbf4aff00db45843b745ba26150076d7edb40c49681d1dae059b6" Oct 30 01:15:04.918661 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:04.918600 5300 scope.go:122] "RemoveContainer" containerID="c66202252d776d341d31f369046b9c3db0946a4e3bc8f13e5bf85ef25fb51aa9" Oct 30 01:15:04.919091 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:04.919062 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh_odh-ai-gateway-infra(eacc79bd-911f-4694-a1ae-deaca2d2df02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh" podUID="eacc79bd-911f-4694-a1ae-deaca2d2df02" Oct 30 01:15:04.920544 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:04.920525 5300 generic.go:356] "Generic (PLEG): container finished" podID="ee041d48-724e-47ae-a69e-2906033b9ce4" containerID="f370b17fb04b2f822f4d41e47180a50369d2971ce017f295d5b6fa7e7f6613ae" exitCode=60 Oct 30 01:15:04.920625 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:04.920602 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv" event={"ID":"ee041d48-724e-47ae-a69e-2906033b9ce4","Type":"ContainerDied","Data":"f370b17fb04b2f822f4d41e47180a50369d2971ce017f295d5b6fa7e7f6613ae"} Oct 30 01:15:04.920942 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:04.920917 5300 scope.go:122] "RemoveContainer" containerID="f370b17fb04b2f822f4d41e47180a50369d2971ce017f295d5b6fa7e7f6613ae" Oct 30 01:15:04.921361 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:04.921335 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv_odh-ai-gateway-infra(ee041d48-724e-47ae-a69e-2906033b9ce4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv" podUID="ee041d48-724e-47ae-a69e-2906033b9ce4" Oct 30 01:15:04.922625 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:04.922605 5300 generic.go:356] "Generic (PLEG): container finished" podID="5abe0803-281e-425c-8f7d-1c99c84c264f" containerID="725140932cffe7c649c8dc2e864be35a46ae2c9745a87dc802e832ae8e33e16c" exitCode=60 Oct 30 01:15:04.922720 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:04.922695 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld" event={"ID":"5abe0803-281e-425c-8f7d-1c99c84c264f","Type":"ContainerDied","Data":"725140932cffe7c649c8dc2e864be35a46ae2c9745a87dc802e832ae8e33e16c"} Oct 30 01:15:04.923010 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:04.922988 5300 scope.go:122] "RemoveContainer" containerID="725140932cffe7c649c8dc2e864be35a46ae2c9745a87dc802e832ae8e33e16c" Oct 30 01:15:04.923365 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:04.923339 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld_odh-ai-gateway-infra(5abe0803-281e-425c-8f7d-1c99c84c264f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld" podUID="5abe0803-281e-425c-8f7d-1c99c84c264f" Oct 30 01:15:04.924685 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:04.924662 5300 generic.go:356] "Generic (PLEG): container finished" podID="7057bdfa-7b6b-4183-bda8-345f355c8567" containerID="c89c970f499bb369d8fc474918768883b88336f78469ed98fbb3f737ec2656d6" exitCode=60 Oct 30 01:15:04.924796 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:04.924756 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj" event={"ID":"7057bdfa-7b6b-4183-bda8-345f355c8567","Type":"ContainerDied","Data":"c89c970f499bb369d8fc474918768883b88336f78469ed98fbb3f737ec2656d6"} Oct 30 01:15:04.925294 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:04.925273 5300 scope.go:122] "RemoveContainer" containerID="c89c970f499bb369d8fc474918768883b88336f78469ed98fbb3f737ec2656d6" Oct 30 01:15:04.925721 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:04.925695 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj_odh-ai-gateway-infra(7057bdfa-7b6b-4183-bda8-345f355c8567)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj" podUID="7057bdfa-7b6b-4183-bda8-345f355c8567" Oct 30 01:15:04.927976 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:04.927952 5300 generic.go:356] "Generic (PLEG): container finished" podID="70bc1eab-1e25-4afb-b6c2-6697d1a77c21" containerID="e3a6531d67b163b4882142a56e55f3d35ae42d53f5e6482df2bab0b86cc7a998" exitCode=60 Oct 30 01:15:04.928129 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:04.928064 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t" event={"ID":"70bc1eab-1e25-4afb-b6c2-6697d1a77c21","Type":"ContainerDied","Data":"e3a6531d67b163b4882142a56e55f3d35ae42d53f5e6482df2bab0b86cc7a998"} Oct 30 01:15:04.928343 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:04.928318 5300 scope.go:122] "RemoveContainer" containerID="e3a6531d67b163b4882142a56e55f3d35ae42d53f5e6482df2bab0b86cc7a998" Oct 30 01:15:04.928718 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:04.928683 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t_odh-ai-gateway-infra(70bc1eab-1e25-4afb-b6c2-6697d1a77c21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t" podUID="70bc1eab-1e25-4afb-b6c2-6697d1a77c21" Oct 30 01:15:04.930721 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:04.930697 5300 generic.go:356] "Generic (PLEG): container finished" podID="0dab0f86-2737-433f-a285-c2719d960bcd" containerID="2cf55a0c54ddd792a13be31379208a66e6e27c8db07b1ad2885afe84ebd15267" exitCode=60 Oct 30 01:15:04.930844 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:04.930747 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-b729h" event={"ID":"0dab0f86-2737-433f-a285-c2719d960bcd","Type":"ContainerDied","Data":"2cf55a0c54ddd792a13be31379208a66e6e27c8db07b1ad2885afe84ebd15267"} Oct 30 01:15:04.931190 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:04.931173 5300 scope.go:122] "RemoveContainer" containerID="2cf55a0c54ddd792a13be31379208a66e6e27c8db07b1ad2885afe84ebd15267" Oct 30 01:15:04.931557 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:04.931535 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888715-b729h_odh-ai-gateway-infra(0dab0f86-2737-433f-a285-c2719d960bcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-b729h" podUID="0dab0f86-2737-433f-a285-c2719d960bcd" Oct 30 01:15:04.934799 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:04.934755 5300 scope.go:122] "RemoveContainer" containerID="81633c8e618f9bac8ce3fe143616c485fea76912f91199029e6849728d94c158" Oct 30 01:15:04.952185 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:04.952165 5300 scope.go:122] "RemoveContainer" containerID="91e6bba5a35da1f7a56f04bc4a29a2f6c188a48f203233761fbc82da002dba0b" Oct 30 01:15:04.968082 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:04.968060 5300 scope.go:122] "RemoveContainer" containerID="e590d597c04a5d4baf67082462b286cbec1f71f810cb6825ffa47bef443a7d4d" Oct 30 01:15:04.982257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:04.982235 5300 scope.go:122] "RemoveContainer" containerID="f922ccb20e1331d489dc317e9c8dc4ba309b1e2d617f7aa8c5d4bc6e6decaca4" Oct 30 01:15:04.996993 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:04.996969 5300 scope.go:122] "RemoveContainer" containerID="7e6d8e2f7b521711a27087c490c8bb005a9f342b663cda3e96f645836a02d0f6" Oct 30 01:15:05.940262 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:05.940232 5300 scope.go:122] "RemoveContainer" containerID="e3a6531d67b163b4882142a56e55f3d35ae42d53f5e6482df2bab0b86cc7a998" Oct 30 01:15:05.940682 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:05.940654 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t_odh-ai-gateway-infra(70bc1eab-1e25-4afb-b6c2-6697d1a77c21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t" podUID="70bc1eab-1e25-4afb-b6c2-6697d1a77c21" Oct 30 01:15:05.941956 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:05.941933 5300 scope.go:122] "RemoveContainer" containerID="2cf55a0c54ddd792a13be31379208a66e6e27c8db07b1ad2885afe84ebd15267" Oct 30 01:15:05.942336 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:05.942310 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888715-b729h_odh-ai-gateway-infra(0dab0f86-2737-433f-a285-c2719d960bcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-b729h" podUID="0dab0f86-2737-433f-a285-c2719d960bcd" Oct 30 01:15:05.943573 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:05.943556 5300 scope.go:122] "RemoveContainer" containerID="c66202252d776d341d31f369046b9c3db0946a4e3bc8f13e5bf85ef25fb51aa9" Oct 30 01:15:05.943815 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:05.943793 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh_odh-ai-gateway-infra(eacc79bd-911f-4694-a1ae-deaca2d2df02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh" podUID="eacc79bd-911f-4694-a1ae-deaca2d2df02" Oct 30 01:15:05.945131 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:05.945106 5300 scope.go:122] "RemoveContainer" containerID="f370b17fb04b2f822f4d41e47180a50369d2971ce017f295d5b6fa7e7f6613ae" Oct 30 01:15:05.945387 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:05.945367 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv_odh-ai-gateway-infra(ee041d48-724e-47ae-a69e-2906033b9ce4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv" podUID="ee041d48-724e-47ae-a69e-2906033b9ce4" Oct 30 01:15:05.946648 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:05.946630 5300 scope.go:122] "RemoveContainer" containerID="725140932cffe7c649c8dc2e864be35a46ae2c9745a87dc802e832ae8e33e16c" Oct 30 01:15:05.946970 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:05.946948 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld_odh-ai-gateway-infra(5abe0803-281e-425c-8f7d-1c99c84c264f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld" podUID="5abe0803-281e-425c-8f7d-1c99c84c264f" Oct 30 01:15:05.948156 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:05.948140 5300 scope.go:122] "RemoveContainer" containerID="c89c970f499bb369d8fc474918768883b88336f78469ed98fbb3f737ec2656d6" Oct 30 01:15:05.948349 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:05.948332 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj_odh-ai-gateway-infra(7057bdfa-7b6b-4183-bda8-345f355c8567)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj" podUID="7057bdfa-7b6b-4183-bda8-345f355c8567" Oct 30 01:15:06.464500 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:06.464460 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:08.172637 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:08.172596 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:09.911022 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:09.910979 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:11.709614 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:11.709572 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:12.513395 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:12.513355 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:13.198078 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:13.198033 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:13.353411 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:13.353368 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:13.691928 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:13.691888 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:15.065735 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:15.065700 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:16.637727 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:16.637692 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:16.813961 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:16.813928 5300 scope.go:122] "RemoveContainer" containerID="725140932cffe7c649c8dc2e864be35a46ae2c9745a87dc802e832ae8e33e16c" Oct 30 01:15:17.170513 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:17.170477 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld" event={"ID":"5abe0803-281e-425c-8f7d-1c99c84c264f","Type":"ContainerStarted","Data":"d6f18cbd2fd4eceea3400bf222bc5af3430ff12c0df0a59679dded6138aad4e4"} Oct 30 01:15:17.813988 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:17.813950 5300 scope.go:122] "RemoveContainer" containerID="c66202252d776d341d31f369046b9c3db0946a4e3bc8f13e5bf85ef25fb51aa9" Oct 30 01:15:17.814501 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:17.814118 5300 scope.go:122] "RemoveContainer" containerID="f370b17fb04b2f822f4d41e47180a50369d2971ce017f295d5b6fa7e7f6613ae" Oct 30 01:15:17.814501 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:17.814295 5300 scope.go:122] "RemoveContainer" containerID="e3a6531d67b163b4882142a56e55f3d35ae42d53f5e6482df2bab0b86cc7a998" Oct 30 01:15:18.180862 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:18.180813 5300 generic.go:356] "Generic (PLEG): container finished" podID="5abe0803-281e-425c-8f7d-1c99c84c264f" containerID="d6f18cbd2fd4eceea3400bf222bc5af3430ff12c0df0a59679dded6138aad4e4" exitCode=60 Oct 30 01:15:18.181058 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:18.180884 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld" event={"ID":"5abe0803-281e-425c-8f7d-1c99c84c264f","Type":"ContainerDied","Data":"d6f18cbd2fd4eceea3400bf222bc5af3430ff12c0df0a59679dded6138aad4e4"} Oct 30 01:15:18.181058 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:18.180945 5300 scope.go:122] "RemoveContainer" containerID="725140932cffe7c649c8dc2e864be35a46ae2c9745a87dc802e832ae8e33e16c" Oct 30 01:15:18.181412 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:18.181395 5300 scope.go:122] "RemoveContainer" containerID="d6f18cbd2fd4eceea3400bf222bc5af3430ff12c0df0a59679dded6138aad4e4" Oct 30 01:15:18.181849 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:18.181826 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld_odh-ai-gateway-infra(5abe0803-281e-425c-8f7d-1c99c84c264f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld" podUID="5abe0803-281e-425c-8f7d-1c99c84c264f" Oct 30 01:15:18.224297 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:18.224266 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld"] Oct 30 01:15:18.348478 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:18.348448 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:18.814378 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:18.814298 5300 scope.go:122] "RemoveContainer" containerID="2cf55a0c54ddd792a13be31379208a66e6e27c8db07b1ad2885afe84ebd15267" Oct 30 01:15:18.814895 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:18.814690 5300 scope.go:122] "RemoveContainer" containerID="c89c970f499bb369d8fc474918768883b88336f78469ed98fbb3f737ec2656d6" Oct 30 01:15:19.191617 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:19.191580 5300 generic.go:356] "Generic (PLEG): container finished" podID="70bc1eab-1e25-4afb-b6c2-6697d1a77c21" containerID="1fa59980af0777b93a29156f4a7abfac38d5cc34413cc8e98b8801d1035a34c5" exitCode=60 Oct 30 01:15:19.191818 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:19.191652 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t" event={"ID":"70bc1eab-1e25-4afb-b6c2-6697d1a77c21","Type":"ContainerDied","Data":"1fa59980af0777b93a29156f4a7abfac38d5cc34413cc8e98b8801d1035a34c5"} Oct 30 01:15:19.191818 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:19.191713 5300 scope.go:122] "RemoveContainer" containerID="e3a6531d67b163b4882142a56e55f3d35ae42d53f5e6482df2bab0b86cc7a998" Oct 30 01:15:19.192290 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:19.192264 5300 scope.go:122] "RemoveContainer" containerID="1fa59980af0777b93a29156f4a7abfac38d5cc34413cc8e98b8801d1035a34c5" Oct 30 01:15:19.192786 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:19.192705 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t_odh-ai-gateway-infra(70bc1eab-1e25-4afb-b6c2-6697d1a77c21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t" podUID="70bc1eab-1e25-4afb-b6c2-6697d1a77c21" Oct 30 01:15:19.194013 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:19.193992 5300 generic.go:356] "Generic (PLEG): container finished" podID="eacc79bd-911f-4694-a1ae-deaca2d2df02" containerID="c00577755e5bc3e9a53521763e0fbaa9747cf69b6cd1da634b66cb18b5b0bfd8" exitCode=60 Oct 30 01:15:19.194116 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:19.194067 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh" event={"ID":"eacc79bd-911f-4694-a1ae-deaca2d2df02","Type":"ContainerDied","Data":"c00577755e5bc3e9a53521763e0fbaa9747cf69b6cd1da634b66cb18b5b0bfd8"} Oct 30 01:15:19.194522 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:19.194505 5300 scope.go:122] "RemoveContainer" containerID="c00577755e5bc3e9a53521763e0fbaa9747cf69b6cd1da634b66cb18b5b0bfd8" Oct 30 01:15:19.194913 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:19.194890 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh_odh-ai-gateway-infra(eacc79bd-911f-4694-a1ae-deaca2d2df02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh" podUID="eacc79bd-911f-4694-a1ae-deaca2d2df02" Oct 30 01:15:19.195870 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:19.195850 5300 generic.go:356] "Generic (PLEG): container finished" podID="ee041d48-724e-47ae-a69e-2906033b9ce4" containerID="18f1270e68d159b9dc907d86dc61c3ed8d2c37ae27ee5d1ff5d26f849ab713c5" exitCode=60 Oct 30 01:15:19.195967 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:19.195924 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv" event={"ID":"ee041d48-724e-47ae-a69e-2906033b9ce4","Type":"ContainerDied","Data":"18f1270e68d159b9dc907d86dc61c3ed8d2c37ae27ee5d1ff5d26f849ab713c5"} Oct 30 01:15:19.196354 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:19.196336 5300 scope.go:122] "RemoveContainer" containerID="18f1270e68d159b9dc907d86dc61c3ed8d2c37ae27ee5d1ff5d26f849ab713c5" Oct 30 01:15:19.196714 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:19.196689 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv_odh-ai-gateway-infra(ee041d48-724e-47ae-a69e-2906033b9ce4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv" podUID="ee041d48-724e-47ae-a69e-2906033b9ce4" Oct 30 01:15:19.205394 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:19.205376 5300 scope.go:122] "RemoveContainer" containerID="c66202252d776d341d31f369046b9c3db0946a4e3bc8f13e5bf85ef25fb51aa9" Oct 30 01:15:19.219332 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:19.219313 5300 scope.go:122] "RemoveContainer" containerID="f370b17fb04b2f822f4d41e47180a50369d2971ce017f295d5b6fa7e7f6613ae" Oct 30 01:15:19.404360 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:19.404333 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld" Oct 30 01:15:19.407427 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:19.407395 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5abe0803-281e-425c-8f7d-1c99c84c264f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5abe0803-281e-425c-8f7d-1c99c84c264f-maas-api-service-ca\") pod \"5abe0803-281e-425c-8f7d-1c99c84c264f\" (UID: \"5abe0803-281e-425c-8f7d-1c99c84c264f\") " Oct 30 01:15:19.407663 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:19.407641 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5abe0803-281e-425c-8f7d-1c99c84c264f-kube-api-access-46xs5\" (UniqueName: \"kubernetes.io/projected/5abe0803-281e-425c-8f7d-1c99c84c264f-kube-api-access-46xs5\") pod \"5abe0803-281e-425c-8f7d-1c99c84c264f\" (UID: \"5abe0803-281e-425c-8f7d-1c99c84c264f\") " Oct 30 01:15:19.407776 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:19.407738 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5abe0803-281e-425c-8f7d-1c99c84c264f-maas-api-service-ca" pod "5abe0803-281e-425c-8f7d-1c99c84c264f" (UID: "5abe0803-281e-425c-8f7d-1c99c84c264f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:19.408401 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:19.408377 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5abe0803-281e-425c-8f7d-1c99c84c264f-maas-api-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:15:19.410187 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:19.410166 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5abe0803-281e-425c-8f7d-1c99c84c264f-kube-api-access-46xs5" pod "5abe0803-281e-425c-8f7d-1c99c84c264f" (UID: "5abe0803-281e-425c-8f7d-1c99c84c264f"). InnerVolumeSpecName "kube-api-access-46xs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:19.509428 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:19.509386 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46xs5\" (UniqueName: \"kubernetes.io/projected/5abe0803-281e-425c-8f7d-1c99c84c264f-kube-api-access-46xs5\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:15:20.088603 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.088517 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:20.214670 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.214637 5300 generic.go:356] "Generic (PLEG): container finished" podID="0dab0f86-2737-433f-a285-c2719d960bcd" containerID="8abe113ae90ac803a83a1c63289e3833ef9275f59543820a1e7a48d1a2f5c669" exitCode=60 Oct 30 01:15:20.214859 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.214713 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-b729h" event={"ID":"0dab0f86-2737-433f-a285-c2719d960bcd","Type":"ContainerDied","Data":"8abe113ae90ac803a83a1c63289e3833ef9275f59543820a1e7a48d1a2f5c669"} Oct 30 01:15:20.214956 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.214858 5300 scope.go:122] "RemoveContainer" containerID="2cf55a0c54ddd792a13be31379208a66e6e27c8db07b1ad2885afe84ebd15267" Oct 30 01:15:20.215457 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.215430 5300 scope.go:122] "RemoveContainer" containerID="8abe113ae90ac803a83a1c63289e3833ef9275f59543820a1e7a48d1a2f5c669" Oct 30 01:15:20.216023 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:20.215988 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888715-b729h_odh-ai-gateway-infra(0dab0f86-2737-433f-a285-c2719d960bcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-b729h" podUID="0dab0f86-2737-433f-a285-c2719d960bcd" Oct 30 01:15:20.230584 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.230541 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld" event={"ID":"5abe0803-281e-425c-8f7d-1c99c84c264f","Type":"ContainerDied","Data":"0fff320956427be4177bbd599018df34f6106852513cc82136f41e36dc5f1acc"} Oct 30 01:15:20.230789 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.230688 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld" Oct 30 01:15:20.233121 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.233088 5300 generic.go:356] "Generic (PLEG): container finished" podID="7057bdfa-7b6b-4183-bda8-345f355c8567" containerID="7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b" exitCode=60 Oct 30 01:15:20.233250 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.233187 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj" event={"ID":"7057bdfa-7b6b-4183-bda8-345f355c8567","Type":"ContainerDied","Data":"7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b"} Oct 30 01:15:20.233671 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.233651 5300 scope.go:122] "RemoveContainer" containerID="7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b" Oct 30 01:15:20.234140 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:20.234105 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj_odh-ai-gateway-infra(7057bdfa-7b6b-4183-bda8-345f355c8567)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj" podUID="7057bdfa-7b6b-4183-bda8-345f355c8567" Oct 30 01:15:20.240988 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.240964 5300 scope.go:122] "RemoveContainer" containerID="d6f18cbd2fd4eceea3400bf222bc5af3430ff12c0df0a59679dded6138aad4e4" Oct 30 01:15:20.252405 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.252380 5300 scope.go:122] "RemoveContainer" containerID="c89c970f499bb369d8fc474918768883b88336f78469ed98fbb3f737ec2656d6" Oct 30 01:15:20.259529 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.259505 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t"] Oct 30 01:15:20.313425 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.311640 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv"] Oct 30 01:15:20.372362 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.371725 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld"] Oct 30 01:15:20.387944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.386174 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888715-bnlld"] Oct 30 01:15:20.412853 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.412811 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh"] Oct 30 01:15:20.477129 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.477100 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t" Oct 30 01:15:20.525684 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.525645 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70bc1eab-1e25-4afb-b6c2-6697d1a77c21-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70bc1eab-1e25-4afb-b6c2-6697d1a77c21-maas-api-service-ca\") pod \"70bc1eab-1e25-4afb-b6c2-6697d1a77c21\" (UID: \"70bc1eab-1e25-4afb-b6c2-6697d1a77c21\") " Oct 30 01:15:20.526263 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.526240 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70bc1eab-1e25-4afb-b6c2-6697d1a77c21-kube-api-access-whvp6\" (UniqueName: \"kubernetes.io/projected/70bc1eab-1e25-4afb-b6c2-6697d1a77c21-kube-api-access-whvp6\") pod \"70bc1eab-1e25-4afb-b6c2-6697d1a77c21\" (UID: \"70bc1eab-1e25-4afb-b6c2-6697d1a77c21\") " Oct 30 01:15:20.527320 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.527271 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70bc1eab-1e25-4afb-b6c2-6697d1a77c21-maas-api-service-ca" pod "70bc1eab-1e25-4afb-b6c2-6697d1a77c21" (UID: "70bc1eab-1e25-4afb-b6c2-6697d1a77c21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:20.530653 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.530479 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70bc1eab-1e25-4afb-b6c2-6697d1a77c21-kube-api-access-whvp6" pod "70bc1eab-1e25-4afb-b6c2-6697d1a77c21" (UID: "70bc1eab-1e25-4afb-b6c2-6697d1a77c21"). InnerVolumeSpecName "kube-api-access-whvp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:20.565602 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.565567 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv" Oct 30 01:15:20.627624 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.627537 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee041d48-724e-47ae-a69e-2906033b9ce4-kube-api-access-s4hsw\" (UniqueName: \"kubernetes.io/projected/ee041d48-724e-47ae-a69e-2906033b9ce4-kube-api-access-s4hsw\") pod \"ee041d48-724e-47ae-a69e-2906033b9ce4\" (UID: \"ee041d48-724e-47ae-a69e-2906033b9ce4\") " Oct 30 01:15:20.627822 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.627686 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee041d48-724e-47ae-a69e-2906033b9ce4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee041d48-724e-47ae-a69e-2906033b9ce4-maas-api-service-ca\") pod \"ee041d48-724e-47ae-a69e-2906033b9ce4\" (UID: \"ee041d48-724e-47ae-a69e-2906033b9ce4\") " Oct 30 01:15:20.628046 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.628020 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee041d48-724e-47ae-a69e-2906033b9ce4-maas-api-service-ca" pod "ee041d48-724e-47ae-a69e-2906033b9ce4" (UID: "ee041d48-724e-47ae-a69e-2906033b9ce4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:20.628833 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.628738 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee041d48-724e-47ae-a69e-2906033b9ce4-maas-api-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:15:20.628833 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.628779 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70bc1eab-1e25-4afb-b6c2-6697d1a77c21-maas-api-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:15:20.628833 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.628793 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whvp6\" (UniqueName: \"kubernetes.io/projected/70bc1eab-1e25-4afb-b6c2-6697d1a77c21-kube-api-access-whvp6\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:15:20.629833 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.629810 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee041d48-724e-47ae-a69e-2906033b9ce4-kube-api-access-s4hsw" pod "ee041d48-724e-47ae-a69e-2906033b9ce4" (UID: "ee041d48-724e-47ae-a69e-2906033b9ce4"). InnerVolumeSpecName "kube-api-access-s4hsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:20.630387 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.630366 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh" Oct 30 01:15:20.730292 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.730250 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eacc79bd-911f-4694-a1ae-deaca2d2df02-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eacc79bd-911f-4694-a1ae-deaca2d2df02-maas-api-service-ca\") pod \"eacc79bd-911f-4694-a1ae-deaca2d2df02\" (UID: \"eacc79bd-911f-4694-a1ae-deaca2d2df02\") " Oct 30 01:15:20.730509 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.730417 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eacc79bd-911f-4694-a1ae-deaca2d2df02-kube-api-access-tkdbz\" (UniqueName: \"kubernetes.io/projected/eacc79bd-911f-4694-a1ae-deaca2d2df02-kube-api-access-tkdbz\") pod \"eacc79bd-911f-4694-a1ae-deaca2d2df02\" (UID: \"eacc79bd-911f-4694-a1ae-deaca2d2df02\") " Oct 30 01:15:20.730692 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.730657 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eacc79bd-911f-4694-a1ae-deaca2d2df02-maas-api-service-ca" pod "eacc79bd-911f-4694-a1ae-deaca2d2df02" (UID: "eacc79bd-911f-4694-a1ae-deaca2d2df02"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:20.731374 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.731348 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4hsw\" (UniqueName: \"kubernetes.io/projected/ee041d48-724e-47ae-a69e-2906033b9ce4-kube-api-access-s4hsw\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:15:20.731469 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.731379 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eacc79bd-911f-4694-a1ae-deaca2d2df02-maas-api-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:15:20.732953 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.732931 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eacc79bd-911f-4694-a1ae-deaca2d2df02-kube-api-access-tkdbz" pod "eacc79bd-911f-4694-a1ae-deaca2d2df02" (UID: "eacc79bd-911f-4694-a1ae-deaca2d2df02"). InnerVolumeSpecName "kube-api-access-tkdbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:20.833550 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:20.833515 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkdbz\" (UniqueName: \"kubernetes.io/projected/eacc79bd-911f-4694-a1ae-deaca2d2df02-kube-api-access-tkdbz\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:15:21.244536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.244448 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t" Oct 30 01:15:21.244993 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.244453 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t" event={"ID":"70bc1eab-1e25-4afb-b6c2-6697d1a77c21","Type":"ContainerDied","Data":"48b1d802c944408a226487f9d001bd88843449635be026f881c992288570d020"} Oct 30 01:15:21.244993 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.244592 5300 scope.go:122] "RemoveContainer" containerID="1fa59980af0777b93a29156f4a7abfac38d5cc34413cc8e98b8801d1035a34c5" Oct 30 01:15:21.247728 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.247705 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh" Oct 30 01:15:21.247874 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.247722 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh" event={"ID":"eacc79bd-911f-4694-a1ae-deaca2d2df02","Type":"ContainerDied","Data":"012c7a5b3b727c98e98273e83d1e57e4c2237a7ee31e6b2f9849f8368ebd9eb3"} Oct 30 01:15:21.250289 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.250264 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv" Oct 30 01:15:21.250403 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.250288 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv" event={"ID":"ee041d48-724e-47ae-a69e-2906033b9ce4","Type":"ContainerDied","Data":"f7f880bc391baf659330fe7ab8b3f55b72d61aebc46e7be879ae0766bd69ceed"} Oct 30 01:15:21.257716 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.257694 5300 scope.go:122] "RemoveContainer" containerID="c00577755e5bc3e9a53521763e0fbaa9747cf69b6cd1da634b66cb18b5b0bfd8" Oct 30 01:15:21.267895 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.267879 5300 scope.go:122] "RemoveContainer" containerID="18f1270e68d159b9dc907d86dc61c3ed8d2c37ae27ee5d1ff5d26f849ab713c5" Oct 30 01:15:21.297401 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.297371 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888715-b729h"] Oct 30 01:15:21.341484 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.341448 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t"] Oct 30 01:15:21.346196 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.346169 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e69dee-29888715-qxw5t"] Oct 30 01:15:21.427055 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.427018 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv"] Oct 30 01:15:21.440391 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.440352 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-add1bb-29888715-lgpsv"] Oct 30 01:15:21.477084 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.477050 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh"] Oct 30 01:15:21.483020 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.482991 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-2be695-29888715-g7vmh"] Oct 30 01:15:21.487917 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.487851 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-b729h" Oct 30 01:15:21.492801 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.492744 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj"] Oct 30 01:15:21.546202 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.543357 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dab0f86-2737-433f-a285-c2719d960bcd-kube-api-access-pq5n4\" (UniqueName: \"kubernetes.io/projected/0dab0f86-2737-433f-a285-c2719d960bcd-kube-api-access-pq5n4\") pod \"0dab0f86-2737-433f-a285-c2719d960bcd\" (UID: \"0dab0f86-2737-433f-a285-c2719d960bcd\") " Oct 30 01:15:21.546202 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.543428 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dab0f86-2737-433f-a285-c2719d960bcd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dab0f86-2737-433f-a285-c2719d960bcd-maas-api-service-ca\") pod \"0dab0f86-2737-433f-a285-c2719d960bcd\" (UID: \"0dab0f86-2737-433f-a285-c2719d960bcd\") " Oct 30 01:15:21.546202 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.544809 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dab0f86-2737-433f-a285-c2719d960bcd-maas-api-service-ca" pod "0dab0f86-2737-433f-a285-c2719d960bcd" (UID: "0dab0f86-2737-433f-a285-c2719d960bcd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:21.549068 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.546488 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dab0f86-2737-433f-a285-c2719d960bcd-kube-api-access-pq5n4" pod "0dab0f86-2737-433f-a285-c2719d960bcd" (UID: "0dab0f86-2737-433f-a285-c2719d960bcd"). InnerVolumeSpecName "kube-api-access-pq5n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:21.621753 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:21.621705 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:15:21.646165 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.646124 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq5n4\" (UniqueName: \"kubernetes.io/projected/0dab0f86-2737-433f-a285-c2719d960bcd-kube-api-access-pq5n4\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:15:21.646165 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.646151 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dab0f86-2737-433f-a285-c2719d960bcd-maas-api-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:15:21.700211 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.700182 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj" Oct 30 01:15:21.746926 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.746888 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7057bdfa-7b6b-4183-bda8-345f355c8567-kube-api-access-v59j2\" (UniqueName: \"kubernetes.io/projected/7057bdfa-7b6b-4183-bda8-345f355c8567-kube-api-access-v59j2\") pod \"7057bdfa-7b6b-4183-bda8-345f355c8567\" (UID: \"7057bdfa-7b6b-4183-bda8-345f355c8567\") " Oct 30 01:15:21.747118 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.746964 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7057bdfa-7b6b-4183-bda8-345f355c8567-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7057bdfa-7b6b-4183-bda8-345f355c8567-maas-api-service-ca\") pod \"7057bdfa-7b6b-4183-bda8-345f355c8567\" (UID: \"7057bdfa-7b6b-4183-bda8-345f355c8567\") " Oct 30 01:15:21.747384 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.747357 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7057bdfa-7b6b-4183-bda8-345f355c8567-maas-api-service-ca" pod "7057bdfa-7b6b-4183-bda8-345f355c8567" (UID: "7057bdfa-7b6b-4183-bda8-345f355c8567"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:15:21.747925 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.747897 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7057bdfa-7b6b-4183-bda8-345f355c8567-maas-api-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:15:21.749566 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.749535 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7057bdfa-7b6b-4183-bda8-345f355c8567-kube-api-access-v59j2" pod "7057bdfa-7b6b-4183-bda8-345f355c8567" (UID: "7057bdfa-7b6b-4183-bda8-345f355c8567"). InnerVolumeSpecName "kube-api-access-v59j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:15:21.824677 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.824589 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5abe0803-281e-425c-8f7d-1c99c84c264f" path="/var/lib/kubelet/pods/5abe0803-281e-425c-8f7d-1c99c84c264f/volumes" Oct 30 01:15:21.825146 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.825129 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70bc1eab-1e25-4afb-b6c2-6697d1a77c21" path="/var/lib/kubelet/pods/70bc1eab-1e25-4afb-b6c2-6697d1a77c21/volumes" Oct 30 01:15:21.825550 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.825537 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eacc79bd-911f-4694-a1ae-deaca2d2df02" path="/var/lib/kubelet/pods/eacc79bd-911f-4694-a1ae-deaca2d2df02/volumes" Oct 30 01:15:21.826206 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.826189 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee041d48-724e-47ae-a69e-2906033b9ce4" path="/var/lib/kubelet/pods/ee041d48-724e-47ae-a69e-2906033b9ce4/volumes" Oct 30 01:15:21.849796 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.849393 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v59j2\" (UniqueName: \"kubernetes.io/projected/7057bdfa-7b6b-4183-bda8-345f355c8567-kube-api-access-v59j2\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:15:21.890369 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:21.890330 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:22.258838 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:22.258793 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-b729h" event={"ID":"0dab0f86-2737-433f-a285-c2719d960bcd","Type":"ContainerDied","Data":"c50433629b037638104424dab684e4ea79e001417c697d7dd8dac6c38cbf6324"} Oct 30 01:15:22.259302 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:22.258867 5300 scope.go:122] "RemoveContainer" containerID="8abe113ae90ac803a83a1c63289e3833ef9275f59543820a1e7a48d1a2f5c669" Oct 30 01:15:22.259302 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:22.258801 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888715-b729h" Oct 30 01:15:22.263196 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:22.263170 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj" Oct 30 01:15:22.263339 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:22.263192 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj" event={"ID":"7057bdfa-7b6b-4183-bda8-345f355c8567","Type":"ContainerDied","Data":"6a8abb3b6776e2c2476988ab93d1af8daa736785cf487572b7add4e13d1e84cd"} Oct 30 01:15:22.272201 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:22.272179 5300 scope.go:122] "RemoveContainer" containerID="7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b" Oct 30 01:15:22.317088 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:22.317054 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888715-b729h"] Oct 30 01:15:22.321060 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:22.321037 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888715-b729h"] Oct 30 01:15:22.360774 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:22.360731 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj"] Oct 30 01:15:22.364533 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:22.364505 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ba3d7b-29888715-4bppj"] Oct 30 01:15:23.735989 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:23.735953 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:23.825729 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:23.825694 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dab0f86-2737-433f-a285-c2719d960bcd" path="/var/lib/kubelet/pods/0dab0f86-2737-433f-a285-c2719d960bcd/volumes" Oct 30 01:15:23.826510 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:23.826480 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7057bdfa-7b6b-4183-bda8-345f355c8567" path="/var/lib/kubelet/pods/7057bdfa-7b6b-4183-bda8-345f355c8567/volumes" Oct 30 01:15:25.617687 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:25.617600 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:25.694821 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:25.694569 5300 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057bdfa_7b6b_4183_bda8_345f355c8567.slice/crio-conmon-7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057bdfa_7b6b_4183_bda8_345f355c8567.slice/crio-7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 01:15:25.965427 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:25.965296 5300 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057bdfa_7b6b_4183_bda8_345f355c8567.slice/crio-conmon-7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057bdfa_7b6b_4183_bda8_345f355c8567.slice/crio-7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 01:15:25.965427 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:25.965316 5300 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057bdfa_7b6b_4183_bda8_345f355c8567.slice/crio-conmon-7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057bdfa_7b6b_4183_bda8_345f355c8567.slice/crio-7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 01:15:27.183345 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:27.183297 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:28.914708 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:28.914667 5300 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057bdfa_7b6b_4183_bda8_345f355c8567.slice/crio-7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057bdfa_7b6b_4183_bda8_345f355c8567.slice/crio-conmon-7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 01:15:28.931742 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:28.931712 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:30.556100 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:30.556066 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:32.235370 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:32.235335 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:33.324704 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:33.324636 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:34.075755 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:34.075717 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:34.824137 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:34.824106 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:34.925121 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:34.925072 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:35.669577 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:35.669540 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:35.843265 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:35.843222 5300 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057bdfa_7b6b_4183_bda8_345f355c8567.slice/crio-7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057bdfa_7b6b_4183_bda8_345f355c8567.slice/crio-conmon-7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 01:15:37.507658 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:37.507624 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:39.406648 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:39.406611 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:41.049722 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:41.049672 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:42.774563 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:42.774528 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:43.992355 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:43.992313 5300 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057bdfa_7b6b_4183_bda8_345f355c8567.slice/crio-conmon-7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057bdfa_7b6b_4183_bda8_345f355c8567.slice/crio-7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 01:15:44.322934 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:44.322837 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:44.419147 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:44.419110 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:45.863985 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:45.863947 5300 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057bdfa_7b6b_4183_bda8_345f355c8567.slice/crio-conmon-7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057bdfa_7b6b_4183_bda8_345f355c8567.slice/crio-7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 01:15:46.018711 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:46.018675 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:47.781717 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:47.781683 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:49.390289 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:49.390237 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:51.254963 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:51.254923 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:51.595183 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:51.595086 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:15:52.836664 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:52.836630 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:54.536377 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:54.536325 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:55.953136 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:55.953091 5300 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057bdfa_7b6b_4183_bda8_345f355c8567.slice/crio-7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057bdfa_7b6b_4183_bda8_345f355c8567.slice/crio-conmon-7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 01:15:56.351044 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:56.351010 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:57.927495 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:57.927453 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:15:58.926307 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:15:58.926262 5300 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057bdfa_7b6b_4183_bda8_345f355c8567.slice/crio-conmon-7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057bdfa_7b6b_4183_bda8_345f355c8567.slice/crio-7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 01:15:59.541371 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:15:59.541332 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:01.374184 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:01.374146 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:03.134583 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:03.134527 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:04.785674 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:04.785642 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:06.043433 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:16:06.043391 5300 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057bdfa_7b6b_4183_bda8_345f355c8567.slice/crio-conmon-7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057bdfa_7b6b_4183_bda8_345f355c8567.slice/crio-7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 01:16:06.652597 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:06.652539 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:08.252046 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:08.252003 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:09.837938 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:09.837903 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:11.640512 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:11.640465 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:13.277292 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:13.277256 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:13.382116 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:13.381955 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:14.006197 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:16:14.006153 5300 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057bdfa_7b6b_4183_bda8_345f355c8567.slice/crio-conmon-7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057bdfa_7b6b_4183_bda8_345f355c8567.slice/crio-7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 01:16:14.951108 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:14.951069 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:16.059882 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:16:16.059835 5300 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057bdfa_7b6b_4183_bda8_345f355c8567.slice/crio-conmon-7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7057bdfa_7b6b_4183_bda8_345f355c8567.slice/crio-7bd68d746a3e672a9b75a42d28cf12ba8c0d45a6811786913baa87bce1108d4b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 01:16:16.744262 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:16.744220 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:18.392231 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:18.392196 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:20.118702 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:20.118668 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:21.595416 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:16:21.595379 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:16:21.799768 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:21.799719 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:22.741341 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:22.741305 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:22.840516 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:22.840482 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:23.241015 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:23.240978 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:24.726666 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:24.726632 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:26.323998 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:26.323963 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:26.616316 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:26.616234 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:26.718314 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:26.718277 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:27.972841 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:27.972806 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:29.613943 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:29.613895 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:31.403086 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:31.403004 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:32.891305 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:32.891257 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:33.455153 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:33.455116 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:34.727095 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:34.727043 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:36.435088 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:36.435054 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:37.943772 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:37.943729 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:39.600650 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:39.600615 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:40.958743 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:40.958699 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4"] Oct 30 01:16:40.983680 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:40.983626 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:41.116079 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:41.116044 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:43.022289 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:43.022255 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:44.640002 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:44.639963 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:46.243044 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:46.243006 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:47.913301 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:47.913262 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:49.681163 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:49.681129 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:51.457362 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:51.457323 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:51.601591 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:16:51.601552 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:16:51.825556 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:51.825521 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:51.933574 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:51.932623 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:53.082418 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:53.082379 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:54.646131 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:54.646088 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:55.985797 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:55.985741 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" podUID="66f98a24-bdba-4445-be54-6e30550e21da" containerName="main" containerID="cri-o://0a90f9d9ffc43fb64019dcdf33d304ea6c6fba5e71042dcefc1a7da75a54d7b2" gracePeriod=45 Oct 30 01:16:56.198503 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:56.195873 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:56.378717 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:56.378682 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" Oct 30 01:16:56.486884 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:56.486842 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-dshm\" (UniqueName: \"kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-dshm\") pod \"66f98a24-bdba-4445-be54-6e30550e21da\" (UID: \"66f98a24-bdba-4445-be54-6e30550e21da\") " Oct 30 01:16:56.487108 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:56.487089 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-tmp-dir\") pod \"66f98a24-bdba-4445-be54-6e30550e21da\" (UID: \"66f98a24-bdba-4445-be54-6e30550e21da\") " Oct 30 01:16:56.487181 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:56.487124 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-model-cache\") pod \"66f98a24-bdba-4445-be54-6e30550e21da\" (UID: \"66f98a24-bdba-4445-be54-6e30550e21da\") " Oct 30 01:16:56.487238 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:56.487201 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66f98a24-bdba-4445-be54-6e30550e21da-kube-api-access-85fsc\" (UniqueName: \"kubernetes.io/projected/66f98a24-bdba-4445-be54-6e30550e21da-kube-api-access-85fsc\") pod \"66f98a24-bdba-4445-be54-6e30550e21da\" (UID: \"66f98a24-bdba-4445-be54-6e30550e21da\") " Oct 30 01:16:56.487325 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:56.487295 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66f98a24-bdba-4445-be54-6e30550e21da-tls-certs\" (UniqueName: \"kubernetes.io/secret/66f98a24-bdba-4445-be54-6e30550e21da-tls-certs\") pod \"66f98a24-bdba-4445-be54-6e30550e21da\" (UID: \"66f98a24-bdba-4445-be54-6e30550e21da\") " Oct 30 01:16:56.487390 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:56.487329 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-home\" (UniqueName: \"kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-home\") pod \"66f98a24-bdba-4445-be54-6e30550e21da\" (UID: \"66f98a24-bdba-4445-be54-6e30550e21da\") " Oct 30 01:16:56.487390 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:56.487348 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-tmp-dir" pod "66f98a24-bdba-4445-be54-6e30550e21da" (UID: "66f98a24-bdba-4445-be54-6e30550e21da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:16:56.487487 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:56.487453 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-model-cache" pod "66f98a24-bdba-4445-be54-6e30550e21da" (UID: "66f98a24-bdba-4445-be54-6e30550e21da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:16:56.487667 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:56.487640 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-home" pod "66f98a24-bdba-4445-be54-6e30550e21da" (UID: "66f98a24-bdba-4445-be54-6e30550e21da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:16:56.488234 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:56.488203 5300 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-tmp-dir\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:16:56.488234 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:56.488235 5300 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-model-cache\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:16:56.488430 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:56.488247 5300 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-home\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:16:56.489843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:56.489810 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f98a24-bdba-4445-be54-6e30550e21da-kube-api-access-85fsc" pod "66f98a24-bdba-4445-be54-6e30550e21da" (UID: "66f98a24-bdba-4445-be54-6e30550e21da"). InnerVolumeSpecName "kube-api-access-85fsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:16:56.489950 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:56.489864 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66f98a24-bdba-4445-be54-6e30550e21da-tls-certs" pod "66f98a24-bdba-4445-be54-6e30550e21da" (UID: "66f98a24-bdba-4445-be54-6e30550e21da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:16:56.489950 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:56.489888 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-dshm" pod "66f98a24-bdba-4445-be54-6e30550e21da" (UID: "66f98a24-bdba-4445-be54-6e30550e21da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:16:56.589410 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:56.589371 5300 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66f98a24-bdba-4445-be54-6e30550e21da-dshm\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:16:56.589410 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:56.589404 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85fsc\" (UniqueName: \"kubernetes.io/projected/66f98a24-bdba-4445-be54-6e30550e21da-kube-api-access-85fsc\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:16:56.589410 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:56.589415 5300 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66f98a24-bdba-4445-be54-6e30550e21da-tls-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:16:57.092256 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:57.092219 5300 generic.go:356] "Generic (PLEG): container finished" podID="66f98a24-bdba-4445-be54-6e30550e21da" containerID="0a90f9d9ffc43fb64019dcdf33d304ea6c6fba5e71042dcefc1a7da75a54d7b2" exitCode=0 Oct 30 01:16:57.092707 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:57.092266 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" Oct 30 01:16:57.092707 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:57.092305 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" event={"ID":"66f98a24-bdba-4445-be54-6e30550e21da","Type":"ContainerDied","Data":"0a90f9d9ffc43fb64019dcdf33d304ea6c6fba5e71042dcefc1a7da75a54d7b2"} Oct 30 01:16:57.092707 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:57.092351 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4" event={"ID":"66f98a24-bdba-4445-be54-6e30550e21da","Type":"ContainerDied","Data":"7b72dac311f7aa5d4fab33065c24f39ab23260a89d9095345c8891ed6c112174"} Oct 30 01:16:57.092707 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:57.092373 5300 scope.go:122] "RemoveContainer" containerID="0a90f9d9ffc43fb64019dcdf33d304ea6c6fba5e71042dcefc1a7da75a54d7b2" Oct 30 01:16:57.103586 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:57.103566 5300 scope.go:122] "RemoveContainer" containerID="0a90f9d9ffc43fb64019dcdf33d304ea6c6fba5e71042dcefc1a7da75a54d7b2" Oct 30 01:16:57.103897 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:16:57.103876 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a90f9d9ffc43fb64019dcdf33d304ea6c6fba5e71042dcefc1a7da75a54d7b2\": container with ID starting with 0a90f9d9ffc43fb64019dcdf33d304ea6c6fba5e71042dcefc1a7da75a54d7b2 not found: ID does not exist" containerID="0a90f9d9ffc43fb64019dcdf33d304ea6c6fba5e71042dcefc1a7da75a54d7b2" Oct 30 01:16:57.103946 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:57.103906 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a90f9d9ffc43fb64019dcdf33d304ea6c6fba5e71042dcefc1a7da75a54d7b2"} err="failed to get container status \"0a90f9d9ffc43fb64019dcdf33d304ea6c6fba5e71042dcefc1a7da75a54d7b2\": rpc error: code = NotFound desc = could not find container \"0a90f9d9ffc43fb64019dcdf33d304ea6c6fba5e71042dcefc1a7da75a54d7b2\": container with ID starting with 0a90f9d9ffc43fb64019dcdf33d304ea6c6fba5e71042dcefc1a7da75a54d7b2 not found: ID does not exist" Oct 30 01:16:57.151447 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:57.151406 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4"] Oct 30 01:16:57.165032 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:57.165002 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-687471/test-model-687471-kserve-65dc8ff97c-r84r4"] Oct 30 01:16:57.823686 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:57.823646 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66f98a24-bdba-4445-be54-6e30550e21da" path="/var/lib/kubelet/pods/66f98a24-bdba-4445-be54-6e30550e21da/volumes" Oct 30 01:16:58.041851 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:58.041804 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:16:59.520471 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:16:59.520437 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:00.499740 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:00.499664 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:01.070253 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:01.070199 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:02.749300 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:02.749214 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:04.325213 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:04.325176 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:05.606117 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:05.606080 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:06.252475 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:06.252441 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:07.210374 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:07.210329 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:07.902386 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:07.902346 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:09.536330 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:09.536292 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:10.334234 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:10.334181 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:11.451073 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:11.451041 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:13.036410 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:13.036379 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:14.622563 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:14.622531 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:16.161777 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:16.161699 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:17.151810 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:17.144253 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:17.835365 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:17.835329 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:19.628172 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:19.628134 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:21.044156 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:21.044122 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:21.597042 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:17:21.596999 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:17:22.534321 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:22.534285 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:23.575726 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:23.575692 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft"] Oct 30 01:17:23.576178 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:23.575941 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" podUID="ded1d999-96c4-4114-a626-615fa35853f2" containerName="istio-proxy" containerID="cri-o://e1371e9f1aba1eb8aec5963ef94caac1ebcee0093572c812adcb7c5b1300f01e" gracePeriod=30 Oct 30 01:17:23.765840 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:23.765805 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9"] Oct 30 01:17:23.766065 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:23.766022 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" podUID="e69d76c4-782a-4518-b52a-72c1cfd7797b" containerName="istio-proxy" containerID="cri-o://43c2d13d41d5d2f7e8e3510884705fea3de0052fdb4b7998943f729e38699338" gracePeriod=30 Oct 30 01:17:23.888459 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:23.888352 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt"] Oct 30 01:17:23.888739 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:23.888622 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" podUID="6f99a456-9f20-4d79-950e-a79b016f5043" containerName="istio-proxy" containerID="cri-o://04258f88fb36e4e8afb159aaf5d6994155ebcd5b46fa6b74231c3e9ab7fc0aca" gracePeriod=30 Oct 30 01:17:24.149928 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:24.149836 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm"] Oct 30 01:17:24.150084 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:24.150060 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" podUID="68f9be1d-13be-4b96-b135-0876eeeae452" containerName="istio-proxy" containerID="cri-o://43fab1c20bdc7183c21ece000292fd113e0e1df39d82db9af757b135ba0cb904" gracePeriod=30 Oct 30 01:17:24.360490 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:24.360456 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2"] Oct 30 01:17:24.360696 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:24.360674 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" podUID="8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1" containerName="istio-proxy" containerID="cri-o://b9c48d57f3af691089f54b17ef3068e89ce40d0bcfc69bafd170ef61c6892009" gracePeriod=30 Oct 30 01:17:24.405288 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:24.405195 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:25.906563 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:25.906525 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6"] Oct 30 01:17:25.907048 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:25.906776 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" podUID="0e890163-36d6-42fb-bae0-3140dbe32425" containerName="istio-proxy" containerID="cri-o://f854a3a5665ca3d70242643908783045e1a42e123a9e4512f360c7b4b6dc5f3d" gracePeriod=30 Oct 30 01:17:25.943177 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:25.943133 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:26.809158 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:26.809119 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk"] Oct 30 01:17:26.816484 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:26.816447 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:27.564219 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:27.564175 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:29.091165 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.091001 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:29.097241 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.096967 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:17:29.289097 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.287531 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-workload-certs\") pod \"ded1d999-96c4-4114-a626-615fa35853f2\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " Oct 30 01:17:29.289097 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.287632 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-credential-socket\") pod \"ded1d999-96c4-4114-a626-615fa35853f2\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " Oct 30 01:17:29.289097 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.287667 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ded1d999-96c4-4114-a626-615fa35853f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded1d999-96c4-4114-a626-615fa35853f2-istio-podinfo\") pod \"ded1d999-96c4-4114-a626-615fa35853f2\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " Oct 30 01:17:29.289097 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.287747 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ded1d999-96c4-4114-a626-615fa35853f2-kube-api-access-khkkg\" (UniqueName: \"kubernetes.io/projected/ded1d999-96c4-4114-a626-615fa35853f2-kube-api-access-khkkg\") pod \"ded1d999-96c4-4114-a626-615fa35853f2\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " Oct 30 01:17:29.289097 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.287875 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ded1d999-96c4-4114-a626-615fa35853f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded1d999-96c4-4114-a626-615fa35853f2-istiod-ca-cert\") pod \"ded1d999-96c4-4114-a626-615fa35853f2\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " Oct 30 01:17:29.289097 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.287994 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-workload-socket\") pod \"ded1d999-96c4-4114-a626-615fa35853f2\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " Oct 30 01:17:29.289097 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.288039 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-istio-envoy\") pod \"ded1d999-96c4-4114-a626-615fa35853f2\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " Oct 30 01:17:29.289097 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.288173 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ded1d999-96c4-4114-a626-615fa35853f2-istio-token\" (UniqueName: \"kubernetes.io/projected/ded1d999-96c4-4114-a626-615fa35853f2-istio-token\") pod \"ded1d999-96c4-4114-a626-615fa35853f2\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " Oct 30 01:17:29.289097 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.288201 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-istio-data\") pod \"ded1d999-96c4-4114-a626-615fa35853f2\" (UID: \"ded1d999-96c4-4114-a626-615fa35853f2\") " Oct 30 01:17:29.290840 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.289675 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-istio-data" pod "ded1d999-96c4-4114-a626-615fa35853f2" (UID: "ded1d999-96c4-4114-a626-615fa35853f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.291541 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.291096 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-workload-certs" pod "ded1d999-96c4-4114-a626-615fa35853f2" (UID: "ded1d999-96c4-4114-a626-615fa35853f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.292942 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.291715 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-credential-socket" pod "ded1d999-96c4-4114-a626-615fa35853f2" (UID: "ded1d999-96c4-4114-a626-615fa35853f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.293526 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.293505 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:17:29.295876 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.294509 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-workload-socket" pod "ded1d999-96c4-4114-a626-615fa35853f2" (UID: "ded1d999-96c4-4114-a626-615fa35853f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.296535 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.296499 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded1d999-96c4-4114-a626-615fa35853f2-istiod-ca-cert" pod "ded1d999-96c4-4114-a626-615fa35853f2" (UID: "ded1d999-96c4-4114-a626-615fa35853f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:17:29.325023 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.324966 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded1d999-96c4-4114-a626-615fa35853f2-istio-token" pod "ded1d999-96c4-4114-a626-615fa35853f2" (UID: "ded1d999-96c4-4114-a626-615fa35853f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:29.332379 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.332302 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-istio-envoy" pod "ded1d999-96c4-4114-a626-615fa35853f2" (UID: "ded1d999-96c4-4114-a626-615fa35853f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.333558 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.333507 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ded1d999-96c4-4114-a626-615fa35853f2-istio-podinfo" pod "ded1d999-96c4-4114-a626-615fa35853f2" (UID: "ded1d999-96c4-4114-a626-615fa35853f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:17:29.336579 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.336475 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded1d999-96c4-4114-a626-615fa35853f2-kube-api-access-khkkg" pod "ded1d999-96c4-4114-a626-615fa35853f2" (UID: "ded1d999-96c4-4114-a626-615fa35853f2"). InnerVolumeSpecName "kube-api-access-khkkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:29.398944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.393286 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-envoy\") pod \"e69d76c4-782a-4518-b52a-72c1cfd7797b\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " Oct 30 01:17:29.398944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.393347 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e69d76c4-782a-4518-b52a-72c1cfd7797b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e69d76c4-782a-4518-b52a-72c1cfd7797b-istiod-ca-cert\") pod \"e69d76c4-782a-4518-b52a-72c1cfd7797b\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " Oct 30 01:17:29.398944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.393403 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e69d76c4-782a-4518-b52a-72c1cfd7797b-kube-api-access-xzdrh\" (UniqueName: \"kubernetes.io/projected/e69d76c4-782a-4518-b52a-72c1cfd7797b-kube-api-access-xzdrh\") pod \"e69d76c4-782a-4518-b52a-72c1cfd7797b\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " Oct 30 01:17:29.398944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.393451 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-token\" (UniqueName: \"kubernetes.io/projected/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-token\") pod \"e69d76c4-782a-4518-b52a-72c1cfd7797b\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " Oct 30 01:17:29.398944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.393574 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-credential-socket\") pod \"e69d76c4-782a-4518-b52a-72c1cfd7797b\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " Oct 30 01:17:29.398944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.393664 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-workload-socket\") pod \"e69d76c4-782a-4518-b52a-72c1cfd7797b\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " Oct 30 01:17:29.398944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.393850 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-workload-certs\") pod \"e69d76c4-782a-4518-b52a-72c1cfd7797b\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " Oct 30 01:17:29.398944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.393902 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-data\") pod \"e69d76c4-782a-4518-b52a-72c1cfd7797b\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " Oct 30 01:17:29.398944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.394031 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-podinfo\") pod \"e69d76c4-782a-4518-b52a-72c1cfd7797b\" (UID: \"e69d76c4-782a-4518-b52a-72c1cfd7797b\") " Oct 30 01:17:29.398944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.394756 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ded1d999-96c4-4114-a626-615fa35853f2-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.398944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.394796 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.398944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.394810 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.398944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.394823 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.398944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.394836 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded1d999-96c4-4114-a626-615fa35853f2-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.398944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.394853 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khkkg\" (UniqueName: \"kubernetes.io/projected/ded1d999-96c4-4114-a626-615fa35853f2-kube-api-access-khkkg\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.398944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.394866 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded1d999-96c4-4114-a626-615fa35853f2-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.398944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.394879 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.398944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.394891 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded1d999-96c4-4114-a626-615fa35853f2-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.398944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.396540 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-workload-certs" pod "e69d76c4-782a-4518-b52a-72c1cfd7797b" (UID: "e69d76c4-782a-4518-b52a-72c1cfd7797b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.398944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.397016 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-credential-socket" pod "e69d76c4-782a-4518-b52a-72c1cfd7797b" (UID: "e69d76c4-782a-4518-b52a-72c1cfd7797b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.398944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.397200 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-workload-socket" pod "e69d76c4-782a-4518-b52a-72c1cfd7797b" (UID: "e69d76c4-782a-4518-b52a-72c1cfd7797b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.398944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.397274 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-data" pod "e69d76c4-782a-4518-b52a-72c1cfd7797b" (UID: "e69d76c4-782a-4518-b52a-72c1cfd7797b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.398944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.397421 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e69d76c4-782a-4518-b52a-72c1cfd7797b-istiod-ca-cert" pod "e69d76c4-782a-4518-b52a-72c1cfd7797b" (UID: "e69d76c4-782a-4518-b52a-72c1cfd7797b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:17:29.406624 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.405843 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-token" pod "e69d76c4-782a-4518-b52a-72c1cfd7797b" (UID: "e69d76c4-782a-4518-b52a-72c1cfd7797b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:29.406624 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.405903 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e69d76c4-782a-4518-b52a-72c1cfd7797b-kube-api-access-xzdrh" pod "e69d76c4-782a-4518-b52a-72c1cfd7797b" (UID: "e69d76c4-782a-4518-b52a-72c1cfd7797b"). InnerVolumeSpecName "kube-api-access-xzdrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:29.407317 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.407291 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-envoy" pod "e69d76c4-782a-4518-b52a-72c1cfd7797b" (UID: "e69d76c4-782a-4518-b52a-72c1cfd7797b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.407650 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.407609 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-podinfo" pod "e69d76c4-782a-4518-b52a-72c1cfd7797b" (UID: "e69d76c4-782a-4518-b52a-72c1cfd7797b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:17:29.471771 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.471723 5300 generic.go:356] "Generic (PLEG): container finished" podID="ded1d999-96c4-4114-a626-615fa35853f2" containerID="e1371e9f1aba1eb8aec5963ef94caac1ebcee0093572c812adcb7c5b1300f01e" exitCode=0 Oct 30 01:17:29.472036 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.471799 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" event={"ID":"ded1d999-96c4-4114-a626-615fa35853f2","Type":"ContainerDied","Data":"e1371e9f1aba1eb8aec5963ef94caac1ebcee0093572c812adcb7c5b1300f01e"} Oct 30 01:17:29.472134 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.472081 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" event={"ID":"ded1d999-96c4-4114-a626-615fa35853f2","Type":"ContainerDied","Data":"cb5e388c4fadb6e694f6c15d1d948408a8392a1a40a026e1315a406ceaf4274a"} Oct 30 01:17:29.472134 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.472119 5300 scope.go:122] "RemoveContainer" containerID="e1371e9f1aba1eb8aec5963ef94caac1ebcee0093572c812adcb7c5b1300f01e" Oct 30 01:17:29.472256 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.471852 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft" Oct 30 01:17:29.477866 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.475694 5300 generic.go:356] "Generic (PLEG): container finished" podID="e69d76c4-782a-4518-b52a-72c1cfd7797b" containerID="43c2d13d41d5d2f7e8e3510884705fea3de0052fdb4b7998943f729e38699338" exitCode=0 Oct 30 01:17:29.477866 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.475802 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" Oct 30 01:17:29.477866 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.475935 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" event={"ID":"e69d76c4-782a-4518-b52a-72c1cfd7797b","Type":"ContainerDied","Data":"43c2d13d41d5d2f7e8e3510884705fea3de0052fdb4b7998943f729e38699338"} Oct 30 01:17:29.477866 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.475998 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9" event={"ID":"e69d76c4-782a-4518-b52a-72c1cfd7797b","Type":"ContainerDied","Data":"78149690b555088140e06105f80c616b4da93cb8bd78b0a024a5793e711e92ce"} Oct 30 01:17:29.505286 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.501425 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.505286 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.501460 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.505286 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.501475 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e69d76c4-782a-4518-b52a-72c1cfd7797b-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.505286 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.501490 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzdrh\" (UniqueName: \"kubernetes.io/projected/e69d76c4-782a-4518-b52a-72c1cfd7797b-kube-api-access-xzdrh\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.505286 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.501505 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.505286 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.501519 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.505286 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.501532 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.505286 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.501545 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.505286 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.501560 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e69d76c4-782a-4518-b52a-72c1cfd7797b-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.536019 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.535012 5300 generic.go:356] "Generic (PLEG): container finished" podID="6f99a456-9f20-4d79-950e-a79b016f5043" containerID="04258f88fb36e4e8afb159aaf5d6994155ebcd5b46fa6b74231c3e9ab7fc0aca" exitCode=0 Oct 30 01:17:29.536019 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.535081 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" event={"ID":"6f99a456-9f20-4d79-950e-a79b016f5043","Type":"ContainerDied","Data":"04258f88fb36e4e8afb159aaf5d6994155ebcd5b46fa6b74231c3e9ab7fc0aca"} Oct 30 01:17:29.538004 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.537808 5300 generic.go:356] "Generic (PLEG): container finished" podID="68f9be1d-13be-4b96-b135-0876eeeae452" containerID="43fab1c20bdc7183c21ece000292fd113e0e1df39d82db9af757b135ba0cb904" exitCode=0 Oct 30 01:17:29.538004 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.537898 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" event={"ID":"68f9be1d-13be-4b96-b135-0876eeeae452","Type":"ContainerDied","Data":"43fab1c20bdc7183c21ece000292fd113e0e1df39d82db9af757b135ba0cb904"} Oct 30 01:17:29.558106 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.557992 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 01:17:29.581103 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.580965 5300 scope.go:122] "RemoveContainer" containerID="e1371e9f1aba1eb8aec5963ef94caac1ebcee0093572c812adcb7c5b1300f01e" Oct 30 01:17:29.584750 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:17:29.584716 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1371e9f1aba1eb8aec5963ef94caac1ebcee0093572c812adcb7c5b1300f01e\": container with ID starting with e1371e9f1aba1eb8aec5963ef94caac1ebcee0093572c812adcb7c5b1300f01e not found: ID does not exist" containerID="e1371e9f1aba1eb8aec5963ef94caac1ebcee0093572c812adcb7c5b1300f01e" Oct 30 01:17:29.584958 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.584779 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1371e9f1aba1eb8aec5963ef94caac1ebcee0093572c812adcb7c5b1300f01e"} err="failed to get container status \"e1371e9f1aba1eb8aec5963ef94caac1ebcee0093572c812adcb7c5b1300f01e\": rpc error: code = NotFound desc = could not find container \"e1371e9f1aba1eb8aec5963ef94caac1ebcee0093572c812adcb7c5b1300f01e\": container with ID starting with e1371e9f1aba1eb8aec5963ef94caac1ebcee0093572c812adcb7c5b1300f01e not found: ID does not exist" Oct 30 01:17:29.584958 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.584808 5300 scope.go:122] "RemoveContainer" containerID="43c2d13d41d5d2f7e8e3510884705fea3de0052fdb4b7998943f729e38699338" Oct 30 01:17:29.624014 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.618963 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft"] Oct 30 01:17:29.624014 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.619035 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-p84ft"] Oct 30 01:17:29.658966 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.658524 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9"] Oct 30 01:17:29.665701 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.663186 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-rjsw9"] Oct 30 01:17:29.671807 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.670681 5300 scope.go:122] "RemoveContainer" containerID="43c2d13d41d5d2f7e8e3510884705fea3de0052fdb4b7998943f729e38699338" Oct 30 01:17:29.672676 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:17:29.672647 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43c2d13d41d5d2f7e8e3510884705fea3de0052fdb4b7998943f729e38699338\": container with ID starting with 43c2d13d41d5d2f7e8e3510884705fea3de0052fdb4b7998943f729e38699338 not found: ID does not exist" containerID="43c2d13d41d5d2f7e8e3510884705fea3de0052fdb4b7998943f729e38699338" Oct 30 01:17:29.672963 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.672935 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43c2d13d41d5d2f7e8e3510884705fea3de0052fdb4b7998943f729e38699338"} err="failed to get container status \"43c2d13d41d5d2f7e8e3510884705fea3de0052fdb4b7998943f729e38699338\": rpc error: code = NotFound desc = could not find container \"43c2d13d41d5d2f7e8e3510884705fea3de0052fdb4b7998943f729e38699338\": container with ID starting with 43c2d13d41d5d2f7e8e3510884705fea3de0052fdb4b7998943f729e38699338 not found: ID does not exist" Oct 30 01:17:29.673061 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.673050 5300 scope.go:122] "RemoveContainer" containerID="9405e2ce69ba988f41b1f3156dd2913008d77daf182dd0b6fa89adc2f23f58ba" Oct 30 01:17:29.701262 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.701160 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:17:29.709615 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.709581 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-credential-socket\") pod \"68f9be1d-13be-4b96-b135-0876eeeae452\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " Oct 30 01:17:29.709899 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.709867 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f99a456-9f20-4d79-950e-a79b016f5043-istio-token\" (UniqueName: \"kubernetes.io/projected/6f99a456-9f20-4d79-950e-a79b016f5043-istio-token\") pod \"6f99a456-9f20-4d79-950e-a79b016f5043\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " Oct 30 01:17:29.710083 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.710067 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f99a456-9f20-4d79-950e-a79b016f5043-kube-api-access-bbf29\" (UniqueName: \"kubernetes.io/projected/6f99a456-9f20-4d79-950e-a79b016f5043-kube-api-access-bbf29\") pod \"6f99a456-9f20-4d79-950e-a79b016f5043\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " Oct 30 01:17:29.710210 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.710196 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-istio-envoy\") pod \"6f99a456-9f20-4d79-950e-a79b016f5043\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " Oct 30 01:17:29.710317 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.710302 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f99a456-9f20-4d79-950e-a79b016f5043-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f99a456-9f20-4d79-950e-a79b016f5043-istio-podinfo\") pod \"6f99a456-9f20-4d79-950e-a79b016f5043\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " Oct 30 01:17:29.710483 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.710470 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-istio-data\") pod \"6f99a456-9f20-4d79-950e-a79b016f5043\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " Oct 30 01:17:29.710603 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.710590 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-istio-envoy\") pod \"68f9be1d-13be-4b96-b135-0876eeeae452\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " Oct 30 01:17:29.710700 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.710685 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f9be1d-13be-4b96-b135-0876eeeae452-kube-api-access-8vjkk\" (UniqueName: \"kubernetes.io/projected/68f9be1d-13be-4b96-b135-0876eeeae452-kube-api-access-8vjkk\") pod \"68f9be1d-13be-4b96-b135-0876eeeae452\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " Oct 30 01:17:29.710875 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.710856 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68f9be1d-13be-4b96-b135-0876eeeae452-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f9be1d-13be-4b96-b135-0876eeeae452-istiod-ca-cert\") pod \"68f9be1d-13be-4b96-b135-0876eeeae452\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " Oct 30 01:17:29.711017 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.711003 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-istio-data\") pod \"68f9be1d-13be-4b96-b135-0876eeeae452\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " Oct 30 01:17:29.711134 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.711120 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-workload-certs\") pod \"68f9be1d-13be-4b96-b135-0876eeeae452\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " Oct 30 01:17:29.711255 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.711239 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-workload-socket\") pod \"68f9be1d-13be-4b96-b135-0876eeeae452\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " Oct 30 01:17:29.711376 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.711361 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-credential-socket\") pod \"6f99a456-9f20-4d79-950e-a79b016f5043\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " Oct 30 01:17:29.711542 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.711526 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-workload-socket\") pod \"6f99a456-9f20-4d79-950e-a79b016f5043\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " Oct 30 01:17:29.711717 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.711702 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f9be1d-13be-4b96-b135-0876eeeae452-istio-token\" (UniqueName: \"kubernetes.io/projected/68f9be1d-13be-4b96-b135-0876eeeae452-istio-token\") pod \"68f9be1d-13be-4b96-b135-0876eeeae452\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " Oct 30 01:17:29.711903 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.711884 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68f9be1d-13be-4b96-b135-0876eeeae452-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f9be1d-13be-4b96-b135-0876eeeae452-istio-podinfo\") pod \"68f9be1d-13be-4b96-b135-0876eeeae452\" (UID: \"68f9be1d-13be-4b96-b135-0876eeeae452\") " Oct 30 01:17:29.712049 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.712033 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f99a456-9f20-4d79-950e-a79b016f5043-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f99a456-9f20-4d79-950e-a79b016f5043-istiod-ca-cert\") pod \"6f99a456-9f20-4d79-950e-a79b016f5043\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " Oct 30 01:17:29.712184 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.712170 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-workload-certs\") pod \"6f99a456-9f20-4d79-950e-a79b016f5043\" (UID: \"6f99a456-9f20-4d79-950e-a79b016f5043\") " Oct 30 01:17:29.713497 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.713475 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-workload-certs" pod "6f99a456-9f20-4d79-950e-a79b016f5043" (UID: "6f99a456-9f20-4d79-950e-a79b016f5043"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.713846 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.713828 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-istio-data" pod "6f99a456-9f20-4d79-950e-a79b016f5043" (UID: "6f99a456-9f20-4d79-950e-a79b016f5043"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.728055 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.728024 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-credential-socket" pod "68f9be1d-13be-4b96-b135-0876eeeae452" (UID: "68f9be1d-13be-4b96-b135-0876eeeae452"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.728448 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.728420 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-workload-socket" pod "68f9be1d-13be-4b96-b135-0876eeeae452" (UID: "68f9be1d-13be-4b96-b135-0876eeeae452"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.753029 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.752962 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f99a456-9f20-4d79-950e-a79b016f5043-istiod-ca-cert" pod "6f99a456-9f20-4d79-950e-a79b016f5043" (UID: "6f99a456-9f20-4d79-950e-a79b016f5043"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:17:29.754222 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.754162 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f9be1d-13be-4b96-b135-0876eeeae452-istio-token" pod "68f9be1d-13be-4b96-b135-0876eeeae452" (UID: "68f9be1d-13be-4b96-b135-0876eeeae452"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:29.754949 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.754912 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-workload-socket" pod "6f99a456-9f20-4d79-950e-a79b016f5043" (UID: "6f99a456-9f20-4d79-950e-a79b016f5043"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.758856 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.758584 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f99a456-9f20-4d79-950e-a79b016f5043-kube-api-access-bbf29" pod "6f99a456-9f20-4d79-950e-a79b016f5043" (UID: "6f99a456-9f20-4d79-950e-a79b016f5043"). InnerVolumeSpecName "kube-api-access-bbf29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:29.758856 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.758653 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f99a456-9f20-4d79-950e-a79b016f5043-istio-token" pod "6f99a456-9f20-4d79-950e-a79b016f5043" (UID: "6f99a456-9f20-4d79-950e-a79b016f5043"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:29.758856 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.758706 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f9be1d-13be-4b96-b135-0876eeeae452-kube-api-access-8vjkk" pod "68f9be1d-13be-4b96-b135-0876eeeae452" (UID: "68f9be1d-13be-4b96-b135-0876eeeae452"). InnerVolumeSpecName "kube-api-access-8vjkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:29.758856 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.758803 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f99a456-9f20-4d79-950e-a79b016f5043-istio-podinfo" pod "6f99a456-9f20-4d79-950e-a79b016f5043" (UID: "6f99a456-9f20-4d79-950e-a79b016f5043"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:17:29.767559 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.766936 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-istio-data" pod "68f9be1d-13be-4b96-b135-0876eeeae452" (UID: "68f9be1d-13be-4b96-b135-0876eeeae452"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.767559 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.767342 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-workload-certs" pod "68f9be1d-13be-4b96-b135-0876eeeae452" (UID: "68f9be1d-13be-4b96-b135-0876eeeae452"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.768086 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.768052 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f9be1d-13be-4b96-b135-0876eeeae452-istiod-ca-cert" pod "68f9be1d-13be-4b96-b135-0876eeeae452" (UID: "68f9be1d-13be-4b96-b135-0876eeeae452"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:17:29.771992 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.771956 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68f9be1d-13be-4b96-b135-0876eeeae452-istio-podinfo" pod "68f9be1d-13be-4b96-b135-0876eeeae452" (UID: "68f9be1d-13be-4b96-b135-0876eeeae452"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:17:29.783668 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.780869 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-credential-socket" pod "6f99a456-9f20-4d79-950e-a79b016f5043" (UID: "6f99a456-9f20-4d79-950e-a79b016f5043"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.834562 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.830556 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.834562 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.832813 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vjkk\" (UniqueName: \"kubernetes.io/projected/68f9be1d-13be-4b96-b135-0876eeeae452-kube-api-access-8vjkk\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.834562 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.832845 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f9be1d-13be-4b96-b135-0876eeeae452-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.834562 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.832862 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.834562 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.832878 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.834562 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.832893 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.834562 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.832907 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.834562 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.832920 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.834562 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.832934 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68f9be1d-13be-4b96-b135-0876eeeae452-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.834562 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.832948 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f9be1d-13be-4b96-b135-0876eeeae452-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.834562 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.832960 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f99a456-9f20-4d79-950e-a79b016f5043-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.834562 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.832973 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.834562 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.832988 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.834562 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.833000 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f99a456-9f20-4d79-950e-a79b016f5043-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.834562 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.833014 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbf29\" (UniqueName: \"kubernetes.io/projected/6f99a456-9f20-4d79-950e-a79b016f5043-kube-api-access-bbf29\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.834562 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.833029 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f99a456-9f20-4d79-950e-a79b016f5043-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.853730 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.851109 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ded1d999-96c4-4114-a626-615fa35853f2" path="/var/lib/kubelet/pods/ded1d999-96c4-4114-a626-615fa35853f2/volumes" Oct 30 01:17:29.853730 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.852016 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e69d76c4-782a-4518-b52a-72c1cfd7797b" path="/var/lib/kubelet/pods/e69d76c4-782a-4518-b52a-72c1cfd7797b/volumes" Oct 30 01:17:29.854201 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.854129 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 01:17:29.878062 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.878023 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-istio-envoy" pod "6f99a456-9f20-4d79-950e-a79b016f5043" (UID: "6f99a456-9f20-4d79-950e-a79b016f5043"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.880401 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.880355 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:29.884577 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.884011 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-istio-envoy" pod "68f9be1d-13be-4b96-b135-0876eeeae452" (UID: "68f9be1d-13be-4b96-b135-0876eeeae452"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.934509 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.934462 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istiod-ca-cert\") pod \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " Oct 30 01:17:29.934824 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.934801 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-token\" (UniqueName: \"kubernetes.io/projected/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-token\") pod \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " Oct 30 01:17:29.935005 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.934990 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-workload-certs\") pod \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " Oct 30 01:17:29.935300 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.935271 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istiod-ca-cert" pod "8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1" (UID: "8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:17:29.935452 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.935283 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-podinfo\") pod \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " Oct 30 01:17:29.935635 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.935616 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-credential-socket\") pod \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " Oct 30 01:17:29.935814 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.935795 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-kube-api-access-7r8ln\" (UniqueName: \"kubernetes.io/projected/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-kube-api-access-7r8ln\") pod \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " Oct 30 01:17:29.936001 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.935983 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-envoy\") pod \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " Oct 30 01:17:29.936561 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.936540 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-data\") pod \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " Oct 30 01:17:29.936676 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.936608 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-workload-socket\") pod \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\" (UID: \"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1\") " Oct 30 01:17:29.936874 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.936846 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-data" pod "8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1" (UID: "8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.937170 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.937144 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-workload-socket" pod "8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1" (UID: "8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.937382 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.937352 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-credential-socket" pod "8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1" (UID: "8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.937596 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.937575 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.937678 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.937602 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.937678 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.937616 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.937678 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.937634 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.937678 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.937648 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f99a456-9f20-4d79-950e-a79b016f5043-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.937678 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.937663 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f9be1d-13be-4b96-b135-0876eeeae452-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:29.937972 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.937796 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-workload-certs" pod "8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1" (UID: "8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.943262 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.943203 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-envoy" pod "8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1" (UID: "8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:29.943831 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.943808 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-token" pod "8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1" (UID: "8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:29.946202 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.946172 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-podinfo" pod "8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1" (UID: "8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:17:29.946991 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.946967 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-kube-api-access-7r8ln" pod "8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1" (UID: "8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1"). InnerVolumeSpecName "kube-api-access-7r8ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:29.993051 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:29.993018 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:30.039678 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:30.039635 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:30.039678 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:30.039681 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r8ln\" (UniqueName: \"kubernetes.io/projected/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-kube-api-access-7r8ln\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:30.040042 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:30.039694 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:30.040042 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:30.039706 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:30.040042 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:30.039718 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:30.092592 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:30.092556 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:30.494820 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:30.494786 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:30.549585 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:30.549552 5300 generic.go:356] "Generic (PLEG): container finished" podID="8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1" containerID="b9c48d57f3af691089f54b17ef3068e89ce40d0bcfc69bafd170ef61c6892009" exitCode=0 Oct 30 01:17:30.549838 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:30.549599 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" Oct 30 01:17:30.549838 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:30.549642 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" event={"ID":"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1","Type":"ContainerDied","Data":"b9c48d57f3af691089f54b17ef3068e89ce40d0bcfc69bafd170ef61c6892009"} Oct 30 01:17:30.549838 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:30.549696 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2" event={"ID":"8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1","Type":"ContainerDied","Data":"53468c57ccb3e9f8e0926de89151075512676964ff2fddb9fb9a7dd3ae3c0061"} Oct 30 01:17:30.549838 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:30.549727 5300 scope.go:122] "RemoveContainer" containerID="b9c48d57f3af691089f54b17ef3068e89ce40d0bcfc69bafd170ef61c6892009" Oct 30 01:17:30.551350 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:30.551331 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" Oct 30 01:17:30.551486 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:30.551363 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt" event={"ID":"6f99a456-9f20-4d79-950e-a79b016f5043","Type":"ContainerDied","Data":"6efc9e01e06e01a46c6288ee62b5035bfa812346b623ba40dbf420d721c89722"} Oct 30 01:17:30.552860 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:30.552827 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" event={"ID":"68f9be1d-13be-4b96-b135-0876eeeae452","Type":"ContainerDied","Data":"513f51888a92a15d62b316df9899ddcc806e5d2521764996d941104f00267e5c"} Oct 30 01:17:30.552860 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:30.552842 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm" Oct 30 01:17:30.566353 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:30.566330 5300 scope.go:122] "RemoveContainer" containerID="b9c48d57f3af691089f54b17ef3068e89ce40d0bcfc69bafd170ef61c6892009" Oct 30 01:17:30.566667 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:17:30.566649 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9c48d57f3af691089f54b17ef3068e89ce40d0bcfc69bafd170ef61c6892009\": container with ID starting with b9c48d57f3af691089f54b17ef3068e89ce40d0bcfc69bafd170ef61c6892009 not found: ID does not exist" containerID="b9c48d57f3af691089f54b17ef3068e89ce40d0bcfc69bafd170ef61c6892009" Oct 30 01:17:30.566754 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:30.566677 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9c48d57f3af691089f54b17ef3068e89ce40d0bcfc69bafd170ef61c6892009"} err="failed to get container status \"b9c48d57f3af691089f54b17ef3068e89ce40d0bcfc69bafd170ef61c6892009\": rpc error: code = NotFound desc = could not find container \"b9c48d57f3af691089f54b17ef3068e89ce40d0bcfc69bafd170ef61c6892009\": container with ID starting with b9c48d57f3af691089f54b17ef3068e89ce40d0bcfc69bafd170ef61c6892009 not found: ID does not exist" Oct 30 01:17:30.566754 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:30.566697 5300 scope.go:122] "RemoveContainer" containerID="04258f88fb36e4e8afb159aaf5d6994155ebcd5b46fa6b74231c3e9ab7fc0aca" Oct 30 01:17:30.577393 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:30.577374 5300 scope.go:122] "RemoveContainer" containerID="43fab1c20bdc7183c21ece000292fd113e0e1df39d82db9af757b135ba0cb904" Oct 30 01:17:30.721582 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:30.721543 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2"] Oct 30 01:17:30.735382 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:30.735350 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-xjvb2"] Oct 30 01:17:30.802233 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:30.802140 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm"] Oct 30 01:17:30.822147 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:30.822113 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-rwxrm"] Oct 30 01:17:30.873490 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:30.873454 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt"] Oct 30 01:17:30.879930 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:30.879897 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-82b647-openshift-default-6747d4d57c-9cjbt"] Oct 30 01:17:31.469801 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.469750 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4zkk6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 01:17:31.470206 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.469825 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" podUID="0e890163-36d6-42fb-bae0-3140dbe32425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 01:17:31.568625 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.568585 5300 generic.go:356] "Generic (PLEG): container finished" podID="0e890163-36d6-42fb-bae0-3140dbe32425" containerID="f854a3a5665ca3d70242643908783045e1a42e123a9e4512f360c7b4b6dc5f3d" exitCode=0 Oct 30 01:17:31.569090 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.568721 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" event={"ID":"0e890163-36d6-42fb-bae0-3140dbe32425","Type":"ContainerDied","Data":"f854a3a5665ca3d70242643908783045e1a42e123a9e4512f360c7b4b6dc5f3d"} Oct 30 01:17:31.660459 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.660427 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 01:17:31.783602 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.783565 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-istio-data\") pod \"0e890163-36d6-42fb-bae0-3140dbe32425\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " Oct 30 01:17:31.783782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.783646 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-workload-socket\") pod \"0e890163-36d6-42fb-bae0-3140dbe32425\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " Oct 30 01:17:31.783782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.783684 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e890163-36d6-42fb-bae0-3140dbe32425-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e890163-36d6-42fb-bae0-3140dbe32425-istio-podinfo\") pod \"0e890163-36d6-42fb-bae0-3140dbe32425\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " Oct 30 01:17:31.783782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.783738 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e890163-36d6-42fb-bae0-3140dbe32425-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e890163-36d6-42fb-bae0-3140dbe32425-istiod-ca-cert\") pod \"0e890163-36d6-42fb-bae0-3140dbe32425\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " Oct 30 01:17:31.783944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.783807 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-istio-envoy\") pod \"0e890163-36d6-42fb-bae0-3140dbe32425\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " Oct 30 01:17:31.783995 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.783939 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-workload-socket" pod "0e890163-36d6-42fb-bae0-3140dbe32425" (UID: "0e890163-36d6-42fb-bae0-3140dbe32425"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:31.783995 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.783962 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-credential-socket\") pod \"0e890163-36d6-42fb-bae0-3140dbe32425\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " Oct 30 01:17:31.784090 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.783947 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-istio-data" pod "0e890163-36d6-42fb-bae0-3140dbe32425" (UID: "0e890163-36d6-42fb-bae0-3140dbe32425"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:31.784187 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.784163 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-credential-socket" pod "0e890163-36d6-42fb-bae0-3140dbe32425" (UID: "0e890163-36d6-42fb-bae0-3140dbe32425"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:31.784311 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.784200 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-workload-certs\") pod \"0e890163-36d6-42fb-bae0-3140dbe32425\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " Oct 30 01:17:31.784311 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.784244 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e890163-36d6-42fb-bae0-3140dbe32425-kube-api-access-27vht\" (UniqueName: \"kubernetes.io/projected/0e890163-36d6-42fb-bae0-3140dbe32425-kube-api-access-27vht\") pod \"0e890163-36d6-42fb-bae0-3140dbe32425\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " Oct 30 01:17:31.784311 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.784267 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e890163-36d6-42fb-bae0-3140dbe32425-istiod-ca-cert" pod "0e890163-36d6-42fb-bae0-3140dbe32425" (UID: "0e890163-36d6-42fb-bae0-3140dbe32425"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:17:31.784311 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.784309 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e890163-36d6-42fb-bae0-3140dbe32425-istio-token\" (UniqueName: \"kubernetes.io/projected/0e890163-36d6-42fb-bae0-3140dbe32425-istio-token\") pod \"0e890163-36d6-42fb-bae0-3140dbe32425\" (UID: \"0e890163-36d6-42fb-bae0-3140dbe32425\") " Oct 30 01:17:31.784535 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.784476 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-workload-certs" pod "0e890163-36d6-42fb-bae0-3140dbe32425" (UID: "0e890163-36d6-42fb-bae0-3140dbe32425"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:31.785083 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.785053 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:31.785083 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.785089 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:31.785261 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.785104 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e890163-36d6-42fb-bae0-3140dbe32425-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:31.785261 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.785118 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:31.785261 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.785129 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:31.786297 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.786255 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e890163-36d6-42fb-bae0-3140dbe32425-istio-podinfo" pod "0e890163-36d6-42fb-bae0-3140dbe32425" (UID: "0e890163-36d6-42fb-bae0-3140dbe32425"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:17:31.786815 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.786786 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e890163-36d6-42fb-bae0-3140dbe32425-kube-api-access-27vht" pod "0e890163-36d6-42fb-bae0-3140dbe32425" (UID: "0e890163-36d6-42fb-bae0-3140dbe32425"). InnerVolumeSpecName "kube-api-access-27vht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:31.786923 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.786901 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-istio-envoy" pod "0e890163-36d6-42fb-bae0-3140dbe32425" (UID: "0e890163-36d6-42fb-bae0-3140dbe32425"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:31.786987 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.786946 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e890163-36d6-42fb-bae0-3140dbe32425-istio-token" pod "0e890163-36d6-42fb-bae0-3140dbe32425" (UID: "0e890163-36d6-42fb-bae0-3140dbe32425"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:31.823048 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.823015 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68f9be1d-13be-4b96-b135-0876eeeae452" path="/var/lib/kubelet/pods/68f9be1d-13be-4b96-b135-0876eeeae452/volumes" Oct 30 01:17:31.823600 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.823584 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f99a456-9f20-4d79-950e-a79b016f5043" path="/var/lib/kubelet/pods/6f99a456-9f20-4d79-950e-a79b016f5043/volumes" Oct 30 01:17:31.824129 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.824113 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1" path="/var/lib/kubelet/pods/8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1/volumes" Oct 30 01:17:31.887383 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.887335 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27vht\" (UniqueName: \"kubernetes.io/projected/0e890163-36d6-42fb-bae0-3140dbe32425-kube-api-access-27vht\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:31.887383 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.887376 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e890163-36d6-42fb-bae0-3140dbe32425-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:31.887383 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.887389 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e890163-36d6-42fb-bae0-3140dbe32425-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:31.887625 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.887400 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e890163-36d6-42fb-bae0-3140dbe32425-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:31.942310 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:31.942277 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:32.578280 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:32.578247 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" Oct 30 01:17:32.578280 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:32.578261 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6" event={"ID":"0e890163-36d6-42fb-bae0-3140dbe32425","Type":"ContainerDied","Data":"d2107d2683d170b396cc91eedef8baf757c7ad5bafd868179df72888504f709c"} Oct 30 01:17:32.578778 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:32.578327 5300 scope.go:122] "RemoveContainer" containerID="f854a3a5665ca3d70242643908783045e1a42e123a9e4512f360c7b4b6dc5f3d" Oct 30 01:17:32.659465 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:32.659423 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6"] Oct 30 01:17:32.671507 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:32.671469 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4zkk6"] Oct 30 01:17:33.533751 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:33.533699 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:33.636551 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:33.636516 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:33.822566 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:33.822484 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e890163-36d6-42fb-bae0-3140dbe32425" path="/var/lib/kubelet/pods/0e890163-36d6-42fb-bae0-3140dbe32425/volumes" Oct 30 01:17:35.031257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:35.031220 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:36.595059 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.595016 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:36.701060 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.701025 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:36.801303 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.801263 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:36.932027 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.931940 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:36.965159 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.965125 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw"] Oct 30 01:17:36.965850 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.965831 5300 cpu_manager.go:414] "containerMap: removing container" podUID="eacc79bd-911f-4694-a1ae-deaca2d2df02" containerName="cleanup" Oct 30 01:17:36.965850 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.965848 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eacc79bd-911f-4694-a1ae-deaca2d2df02" containerName="cleanup" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.965861 5300 cpu_manager.go:414] "containerMap: removing container" podUID="0dab0f86-2737-433f-a285-c2719d960bcd" containerName="cleanup" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.965868 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dab0f86-2737-433f-a285-c2719d960bcd" containerName="cleanup" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.965888 5300 cpu_manager.go:414] "containerMap: removing container" podUID="0dab0f86-2737-433f-a285-c2719d960bcd" containerName="cleanup" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.965893 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dab0f86-2737-433f-a285-c2719d960bcd" containerName="cleanup" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.965901 5300 cpu_manager.go:414] "containerMap: removing container" podUID="7057bdfa-7b6b-4183-bda8-345f355c8567" containerName="cleanup" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.965905 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7057bdfa-7b6b-4183-bda8-345f355c8567" containerName="cleanup" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.965917 5300 cpu_manager.go:414] "containerMap: removing container" podUID="6f99a456-9f20-4d79-950e-a79b016f5043" containerName="istio-proxy" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.965921 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f99a456-9f20-4d79-950e-a79b016f5043" containerName="istio-proxy" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.965930 5300 cpu_manager.go:414] "containerMap: removing container" podUID="66f98a24-bdba-4445-be54-6e30550e21da" containerName="main" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.965935 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66f98a24-bdba-4445-be54-6e30550e21da" containerName="main" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.965942 5300 cpu_manager.go:414] "containerMap: removing container" podUID="ee041d48-724e-47ae-a69e-2906033b9ce4" containerName="cleanup" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.965947 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee041d48-724e-47ae-a69e-2906033b9ce4" containerName="cleanup" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.965962 5300 cpu_manager.go:414] "containerMap: removing container" podUID="0e890163-36d6-42fb-bae0-3140dbe32425" containerName="istio-proxy" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.965967 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e890163-36d6-42fb-bae0-3140dbe32425" containerName="istio-proxy" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.965974 5300 cpu_manager.go:414] "containerMap: removing container" podUID="ee041d48-724e-47ae-a69e-2906033b9ce4" containerName="cleanup" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.965978 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee041d48-724e-47ae-a69e-2906033b9ce4" containerName="cleanup" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.965985 5300 cpu_manager.go:414] "containerMap: removing container" podUID="6f99a456-9f20-4d79-950e-a79b016f5043" containerName="istio-proxy" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.965989 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f99a456-9f20-4d79-950e-a79b016f5043" containerName="istio-proxy" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.965996 5300 cpu_manager.go:414] "containerMap: removing container" podUID="70bc1eab-1e25-4afb-b6c2-6697d1a77c21" containerName="cleanup" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966000 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70bc1eab-1e25-4afb-b6c2-6697d1a77c21" containerName="cleanup" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966009 5300 cpu_manager.go:414] "containerMap: removing container" podUID="e69d76c4-782a-4518-b52a-72c1cfd7797b" containerName="istio-proxy" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966014 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e69d76c4-782a-4518-b52a-72c1cfd7797b" containerName="istio-proxy" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966022 5300 cpu_manager.go:414] "containerMap: removing container" podUID="eacc79bd-911f-4694-a1ae-deaca2d2df02" containerName="cleanup" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966027 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eacc79bd-911f-4694-a1ae-deaca2d2df02" containerName="cleanup" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966032 5300 cpu_manager.go:414] "containerMap: removing container" podUID="ee041d48-724e-47ae-a69e-2906033b9ce4" containerName="cleanup" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966037 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee041d48-724e-47ae-a69e-2906033b9ce4" containerName="cleanup" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966046 5300 cpu_manager.go:414] "containerMap: removing container" podUID="5abe0803-281e-425c-8f7d-1c99c84c264f" containerName="cleanup" Oct 30 01:17:36.966030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966050 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5abe0803-281e-425c-8f7d-1c99c84c264f" containerName="cleanup" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966061 5300 cpu_manager.go:414] "containerMap: removing container" podUID="70bc1eab-1e25-4afb-b6c2-6697d1a77c21" containerName="cleanup" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966065 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70bc1eab-1e25-4afb-b6c2-6697d1a77c21" containerName="cleanup" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966072 5300 cpu_manager.go:414] "containerMap: removing container" podUID="ded1d999-96c4-4114-a626-615fa35853f2" containerName="istio-proxy" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966077 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ded1d999-96c4-4114-a626-615fa35853f2" containerName="istio-proxy" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966085 5300 cpu_manager.go:414] "containerMap: removing container" podUID="70bc1eab-1e25-4afb-b6c2-6697d1a77c21" containerName="cleanup" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966090 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70bc1eab-1e25-4afb-b6c2-6697d1a77c21" containerName="cleanup" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966097 5300 cpu_manager.go:414] "containerMap: removing container" podUID="68f9be1d-13be-4b96-b135-0876eeeae452" containerName="istio-proxy" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966101 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68f9be1d-13be-4b96-b135-0876eeeae452" containerName="istio-proxy" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966109 5300 cpu_manager.go:414] "containerMap: removing container" podUID="5abe0803-281e-425c-8f7d-1c99c84c264f" containerName="cleanup" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966113 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5abe0803-281e-425c-8f7d-1c99c84c264f" containerName="cleanup" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966119 5300 cpu_manager.go:414] "containerMap: removing container" podUID="8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1" containerName="istio-proxy" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966123 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1" containerName="istio-proxy" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966129 5300 cpu_manager.go:414] "containerMap: removing container" podUID="7057bdfa-7b6b-4183-bda8-345f355c8567" containerName="cleanup" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966133 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7057bdfa-7b6b-4183-bda8-345f355c8567" containerName="cleanup" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966146 5300 cpu_manager.go:414] "containerMap: removing container" podUID="7057bdfa-7b6b-4183-bda8-345f355c8567" containerName="cleanup" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966151 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7057bdfa-7b6b-4183-bda8-345f355c8567" containerName="cleanup" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966240 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="70bc1eab-1e25-4afb-b6c2-6697d1a77c21" containerName="cleanup" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966250 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="68f9be1d-13be-4b96-b135-0876eeeae452" containerName="istio-proxy" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966255 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="5abe0803-281e-425c-8f7d-1c99c84c264f" containerName="cleanup" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966260 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="5abe0803-281e-425c-8f7d-1c99c84c264f" containerName="cleanup" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966266 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dff7a64-3ddd-42d6-85d1-1a56cb05bcb1" containerName="istio-proxy" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966272 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="7057bdfa-7b6b-4183-bda8-345f355c8567" containerName="cleanup" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966278 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="7057bdfa-7b6b-4183-bda8-345f355c8567" containerName="cleanup" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966285 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="eacc79bd-911f-4694-a1ae-deaca2d2df02" containerName="cleanup" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966291 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dab0f86-2737-433f-a285-c2719d960bcd" containerName="cleanup" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966299 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dab0f86-2737-433f-a285-c2719d960bcd" containerName="cleanup" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966304 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="7057bdfa-7b6b-4183-bda8-345f355c8567" containerName="cleanup" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966309 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f99a456-9f20-4d79-950e-a79b016f5043" containerName="istio-proxy" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966313 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="66f98a24-bdba-4445-be54-6e30550e21da" containerName="main" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966318 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee041d48-724e-47ae-a69e-2906033b9ce4" containerName="cleanup" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966325 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e890163-36d6-42fb-bae0-3140dbe32425" containerName="istio-proxy" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966329 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee041d48-724e-47ae-a69e-2906033b9ce4" containerName="cleanup" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966334 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="e69d76c4-782a-4518-b52a-72c1cfd7797b" containerName="istio-proxy" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966343 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dab0f86-2737-433f-a285-c2719d960bcd" containerName="cleanup" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966350 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="5abe0803-281e-425c-8f7d-1c99c84c264f" containerName="cleanup" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966357 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="70bc1eab-1e25-4afb-b6c2-6697d1a77c21" containerName="cleanup" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966362 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="eacc79bd-911f-4694-a1ae-deaca2d2df02" containerName="cleanup" Oct 30 01:17:36.967536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.966367 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="ded1d999-96c4-4114-a626-615fa35853f2" containerName="istio-proxy" Oct 30 01:17:36.971672 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.971647 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" Oct 30 01:17:36.996384 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:36.996353 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw"] Oct 30 01:17:37.005170 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:37.005139 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-eacea3\"/\"openshift-service-ca.crt\"" Oct 30 01:17:37.005170 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:37.005149 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-eacea3\"/\"rate-test-model-eacea3-kserve-self-signed-certs\"" Oct 30 01:17:37.005375 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:37.005234 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-eacea3\"/\"kube-root-ca.crt\"" Oct 30 01:17:37.005375 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:37.005294 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-eacea3\"/\"default-dockercfg-tfxjr\"" Oct 30 01:17:37.063310 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:37.063277 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-model-cache\") pod \"rate-test-model-eacea3-kserve-647f6df8b9-kt7jw\" (UID: \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" Oct 30 01:17:37.063518 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:37.063419 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-home\") pod \"rate-test-model-eacea3-kserve-647f6df8b9-kt7jw\" (UID: \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" Oct 30 01:17:37.063518 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:37.063477 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-dshm\") pod \"rate-test-model-eacea3-kserve-647f6df8b9-kt7jw\" (UID: \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" Oct 30 01:17:37.063637 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:37.063589 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vsnw\" (UniqueName: \"kubernetes.io/projected/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-kube-api-access-7vsnw\") pod \"rate-test-model-eacea3-kserve-647f6df8b9-kt7jw\" (UID: \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" Oct 30 01:17:37.063703 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:37.063658 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-tls-certs\") pod \"rate-test-model-eacea3-kserve-647f6df8b9-kt7jw\" (UID: \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" Oct 30 01:17:37.063850 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:37.063834 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-tmp-dir\") pod \"rate-test-model-eacea3-kserve-647f6df8b9-kt7jw\" (UID: \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" Oct 30 01:17:37.099253 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:37.099217 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:37.165583 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:37.165549 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-tmp-dir\") pod \"rate-test-model-eacea3-kserve-647f6df8b9-kt7jw\" (UID: \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" Oct 30 01:17:37.165778 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:37.165631 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-model-cache\") pod \"rate-test-model-eacea3-kserve-647f6df8b9-kt7jw\" (UID: \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" Oct 30 01:17:37.165778 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:37.165743 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-home\") pod \"rate-test-model-eacea3-kserve-647f6df8b9-kt7jw\" (UID: \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" Oct 30 01:17:37.165881 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:37.165832 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-dshm\") pod \"rate-test-model-eacea3-kserve-647f6df8b9-kt7jw\" (UID: \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" Oct 30 01:17:37.165985 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:37.165961 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vsnw\" (UniqueName: \"kubernetes.io/projected/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-kube-api-access-7vsnw\") pod \"rate-test-model-eacea3-kserve-647f6df8b9-kt7jw\" (UID: \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" Oct 30 01:17:37.166063 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:37.166033 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-tls-certs\") pod \"rate-test-model-eacea3-kserve-647f6df8b9-kt7jw\" (UID: \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" Oct 30 01:17:37.166063 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:37.166043 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-tmp-dir\") pod \"rate-test-model-eacea3-kserve-647f6df8b9-kt7jw\" (UID: \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" Oct 30 01:17:37.166169 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:37.166109 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-model-cache\") pod \"rate-test-model-eacea3-kserve-647f6df8b9-kt7jw\" (UID: \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" Oct 30 01:17:37.166169 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:37.166134 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-home\") pod \"rate-test-model-eacea3-kserve-647f6df8b9-kt7jw\" (UID: \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" Oct 30 01:17:37.168102 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:37.168074 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-dshm\") pod \"rate-test-model-eacea3-kserve-647f6df8b9-kt7jw\" (UID: \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" Oct 30 01:17:37.168454 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:37.168433 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-tls-certs\") pod \"rate-test-model-eacea3-kserve-647f6df8b9-kt7jw\" (UID: \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" Oct 30 01:17:37.191855 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:37.191777 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vsnw\" (UniqueName: \"kubernetes.io/projected/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-kube-api-access-7vsnw\") pod \"rate-test-model-eacea3-kserve-647f6df8b9-kt7jw\" (UID: \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\") " pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" Oct 30 01:17:37.301124 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:37.301083 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" Oct 30 01:17:37.651303 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:37.651266 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw"] Oct 30 01:17:37.651707 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:17:37.651510 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee55bf23_0c5c_4078_9149_ff5a6fab2c3d.slice/crio-105e5ac1cb91e62df879e0dc4122f6673f749e79f1fc80f38774dbd9ea625731 WatchSource:0}: Error finding container 105e5ac1cb91e62df879e0dc4122f6673f749e79f1fc80f38774dbd9ea625731: Status 404 returned error can't find the container with id 105e5ac1cb91e62df879e0dc4122f6673f749e79f1fc80f38774dbd9ea625731 Oct 30 01:17:38.184282 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:38.184197 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:38.641880 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:38.641830 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" event={"ID":"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d","Type":"ContainerStarted","Data":"7fb2d755c9a5f4f70d52242ce2527ccfc17f70a648127975e4e1b8106746bc59"} Oct 30 01:17:38.642085 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:38.641892 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" event={"ID":"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d","Type":"ContainerStarted","Data":"105e5ac1cb91e62df879e0dc4122f6673f749e79f1fc80f38774dbd9ea625731"} Oct 30 01:17:38.706953 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:38.706890 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" podStartSLOduration=2.706875899 podStartE2EDuration="2.706875899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:38.704876523 +0000 UTC m=+2059.020744848" watchObservedRunningTime="2026-10-30 01:17:38.706875899 +0000 UTC m=+2059.022744225" Oct 30 01:17:38.918995 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:38.918914 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl"] Oct 30 01:17:38.920003 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:38.919978 5300 cpu_manager.go:414] "containerMap: removing container" podUID="eacc79bd-911f-4694-a1ae-deaca2d2df02" containerName="cleanup" Oct 30 01:17:38.920003 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:38.920002 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eacc79bd-911f-4694-a1ae-deaca2d2df02" containerName="cleanup" Oct 30 01:17:38.920176 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:38.920040 5300 cpu_manager.go:414] "containerMap: removing container" podUID="5abe0803-281e-425c-8f7d-1c99c84c264f" containerName="cleanup" Oct 30 01:17:38.920176 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:38.920049 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5abe0803-281e-425c-8f7d-1c99c84c264f" containerName="cleanup" Oct 30 01:17:38.920176 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:38.920097 5300 cpu_manager.go:414] "containerMap: removing container" podUID="0dab0f86-2737-433f-a285-c2719d960bcd" containerName="cleanup" Oct 30 01:17:38.920176 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:38.920105 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dab0f86-2737-433f-a285-c2719d960bcd" containerName="cleanup" Oct 30 01:17:38.920312 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:38.920248 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f99a456-9f20-4d79-950e-a79b016f5043" containerName="istio-proxy" Oct 30 01:17:38.920312 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:38.920266 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee041d48-724e-47ae-a69e-2906033b9ce4" containerName="cleanup" Oct 30 01:17:38.920312 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:38.920283 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="70bc1eab-1e25-4afb-b6c2-6697d1a77c21" containerName="cleanup" Oct 30 01:17:38.920312 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:38.920299 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="eacc79bd-911f-4694-a1ae-deaca2d2df02" containerName="cleanup" Oct 30 01:17:38.924833 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:38.924809 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" Oct 30 01:17:38.958336 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:38.957673 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-64f472\"/\"kube-root-ca.crt\"" Oct 30 01:17:38.958336 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:38.957683 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-64f472\"/\"sub-test-model-64f472-kserve-self-signed-certs\"" Oct 30 01:17:38.959694 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:38.959665 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl"] Oct 30 01:17:38.960079 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:38.960053 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:39.028174 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:39.028141 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-64f472\"/\"default-dockercfg-66ftk\"" Oct 30 01:17:39.046186 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:39.046158 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-64f472\"/\"openshift-service-ca.crt\"" Oct 30 01:17:39.078478 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:39.078431 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:39.108599 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:39.108553 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-home\") pod \"sub-test-model-64f472-kserve-b5488fb95-xkbfl\" (UID: \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\") " pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" Oct 30 01:17:39.108599 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:39.108600 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt6hl\" (UniqueName: \"kubernetes.io/projected/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-kube-api-access-qt6hl\") pod \"sub-test-model-64f472-kserve-b5488fb95-xkbfl\" (UID: \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\") " pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" Oct 30 01:17:39.108893 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:39.108778 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-tls-certs\") pod \"sub-test-model-64f472-kserve-b5488fb95-xkbfl\" (UID: \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\") " pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" Oct 30 01:17:39.108954 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:39.108890 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-dshm\") pod \"sub-test-model-64f472-kserve-b5488fb95-xkbfl\" (UID: \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\") " pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" Oct 30 01:17:39.109003 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:39.108992 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-model-cache\") pod \"sub-test-model-64f472-kserve-b5488fb95-xkbfl\" (UID: \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\") " pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" Oct 30 01:17:39.109067 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:39.109043 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-tmp-dir\") pod \"sub-test-model-64f472-kserve-b5488fb95-xkbfl\" (UID: \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\") " pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" Oct 30 01:17:39.211805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:39.211673 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-home\") pod \"sub-test-model-64f472-kserve-b5488fb95-xkbfl\" (UID: \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\") " pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" Oct 30 01:17:39.211805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:39.211734 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt6hl\" (UniqueName: \"kubernetes.io/projected/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-kube-api-access-qt6hl\") pod \"sub-test-model-64f472-kserve-b5488fb95-xkbfl\" (UID: \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\") " pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" Oct 30 01:17:39.212050 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:39.211896 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-tls-certs\") pod \"sub-test-model-64f472-kserve-b5488fb95-xkbfl\" (UID: \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\") " pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" Oct 30 01:17:39.212050 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:39.211995 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-dshm\") pod \"sub-test-model-64f472-kserve-b5488fb95-xkbfl\" (UID: \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\") " pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" Oct 30 01:17:39.212050 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:39.212042 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-model-cache\") pod \"sub-test-model-64f472-kserve-b5488fb95-xkbfl\" (UID: \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\") " pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" Oct 30 01:17:39.212228 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:39.212068 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-tmp-dir\") pod \"sub-test-model-64f472-kserve-b5488fb95-xkbfl\" (UID: \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\") " pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" Oct 30 01:17:39.212228 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:39.212189 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-home\") pod \"sub-test-model-64f472-kserve-b5488fb95-xkbfl\" (UID: \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\") " pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" Oct 30 01:17:39.212478 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:39.212455 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-model-cache\") pod \"sub-test-model-64f472-kserve-b5488fb95-xkbfl\" (UID: \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\") " pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" Oct 30 01:17:39.212555 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:39.212466 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-tmp-dir\") pod \"sub-test-model-64f472-kserve-b5488fb95-xkbfl\" (UID: \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\") " pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" Oct 30 01:17:39.214513 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:39.214485 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-dshm\") pod \"sub-test-model-64f472-kserve-b5488fb95-xkbfl\" (UID: \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\") " pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" Oct 30 01:17:39.214749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:39.214724 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-tls-certs\") pod \"sub-test-model-64f472-kserve-b5488fb95-xkbfl\" (UID: \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\") " pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" Oct 30 01:17:39.233508 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:39.233468 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt6hl\" (UniqueName: \"kubernetes.io/projected/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-kube-api-access-qt6hl\") pod \"sub-test-model-64f472-kserve-b5488fb95-xkbfl\" (UID: \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\") " pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" Oct 30 01:17:39.264496 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:39.264449 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" Oct 30 01:17:39.302951 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:39.302888 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" Oct 30 01:17:39.612022 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:39.611991 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl"] Oct 30 01:17:39.612316 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:17:39.612290 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2388217_f8d3_4bd7_9f6a_dc0bfd7db1d7.slice/crio-24a951f3066f0b5613b055ff82622414737054e9ea4653068eaf2928d73c4886 WatchSource:0}: Error finding container 24a951f3066f0b5613b055ff82622414737054e9ea4653068eaf2928d73c4886: Status 404 returned error can't find the container with id 24a951f3066f0b5613b055ff82622414737054e9ea4653068eaf2928d73c4886 Oct 30 01:17:39.650835 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:39.650794 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" event={"ID":"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7","Type":"ContainerStarted","Data":"24a951f3066f0b5613b055ff82622414737054e9ea4653068eaf2928d73c4886"} Oct 30 01:17:40.661297 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:40.661257 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" event={"ID":"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7","Type":"ContainerStarted","Data":"f90d40a21b8ef4968166db4c1d1c637dfc0000d174285abed9c0062b7bfddf67"} Oct 30 01:17:40.724068 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:40.724018 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:41.264570 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:41.264534 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" Oct 30 01:17:41.832455 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:41.832406 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" podUID="26f9928d-0276-4c79-800a-9886a99c6499" containerName="main" containerID="cri-o://e86d4632e9eb84c2bbb3373c31c5d5bce2adaa8155992197eeb4ddb0c13409f1" gracePeriod=45 Oct 30 01:17:42.143092 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.143060 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" Oct 30 01:17:42.185947 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.185895 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" podStartSLOduration=4.185877898 podStartE2EDuration="4.185877898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:40.716607513 +0000 UTC m=+2061.032475840" watchObservedRunningTime="2026-10-30 01:17:42.185877898 +0000 UTC m=+2062.501746201" Oct 30 01:17:42.223087 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.223029 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-model-cache\") pod \"26f9928d-0276-4c79-800a-9886a99c6499\" (UID: \"26f9928d-0276-4c79-800a-9886a99c6499\") " Oct 30 01:17:42.223294 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.223103 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-home\" (UniqueName: \"kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-home\") pod \"26f9928d-0276-4c79-800a-9886a99c6499\" (UID: \"26f9928d-0276-4c79-800a-9886a99c6499\") " Oct 30 01:17:42.223294 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.223185 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-dshm\" (UniqueName: \"kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-dshm\") pod \"26f9928d-0276-4c79-800a-9886a99c6499\" (UID: \"26f9928d-0276-4c79-800a-9886a99c6499\") " Oct 30 01:17:42.223294 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.223251 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f9928d-0276-4c79-800a-9886a99c6499-kube-api-access-5cctx\" (UniqueName: \"kubernetes.io/projected/26f9928d-0276-4c79-800a-9886a99c6499-kube-api-access-5cctx\") pod \"26f9928d-0276-4c79-800a-9886a99c6499\" (UID: \"26f9928d-0276-4c79-800a-9886a99c6499\") " Oct 30 01:17:42.223484 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.223351 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26f9928d-0276-4c79-800a-9886a99c6499-tls-certs\" (UniqueName: \"kubernetes.io/secret/26f9928d-0276-4c79-800a-9886a99c6499-tls-certs\") pod \"26f9928d-0276-4c79-800a-9886a99c6499\" (UID: \"26f9928d-0276-4c79-800a-9886a99c6499\") " Oct 30 01:17:42.223484 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.223419 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-model-cache" pod "26f9928d-0276-4c79-800a-9886a99c6499" (UID: "26f9928d-0276-4c79-800a-9886a99c6499"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:42.223484 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.223437 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-home" pod "26f9928d-0276-4c79-800a-9886a99c6499" (UID: "26f9928d-0276-4c79-800a-9886a99c6499"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:42.223484 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.223464 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-tmp-dir\") pod \"26f9928d-0276-4c79-800a-9886a99c6499\" (UID: \"26f9928d-0276-4c79-800a-9886a99c6499\") " Oct 30 01:17:42.223799 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.223751 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-tmp-dir" pod "26f9928d-0276-4c79-800a-9886a99c6499" (UID: "26f9928d-0276-4c79-800a-9886a99c6499"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:42.224196 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.224151 5300 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-tmp-dir\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:42.224196 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.224181 5300 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-model-cache\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:42.224196 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.224195 5300 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-home\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:42.226181 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.226150 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-dshm" pod "26f9928d-0276-4c79-800a-9886a99c6499" (UID: "26f9928d-0276-4c79-800a-9886a99c6499"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:42.226391 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.226366 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26f9928d-0276-4c79-800a-9886a99c6499-tls-certs" pod "26f9928d-0276-4c79-800a-9886a99c6499" (UID: "26f9928d-0276-4c79-800a-9886a99c6499"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:17:42.226444 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.226407 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f9928d-0276-4c79-800a-9886a99c6499-kube-api-access-5cctx" pod "26f9928d-0276-4c79-800a-9886a99c6499" (UID: "26f9928d-0276-4c79-800a-9886a99c6499"). InnerVolumeSpecName "kube-api-access-5cctx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:42.325559 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.325517 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cctx\" (UniqueName: \"kubernetes.io/projected/26f9928d-0276-4c79-800a-9886a99c6499-kube-api-access-5cctx\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:42.325559 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.325551 5300 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26f9928d-0276-4c79-800a-9886a99c6499-tls-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:42.325559 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.325561 5300 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26f9928d-0276-4c79-800a-9886a99c6499-dshm\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:17:42.411967 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.411870 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:42.677959 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.677859 5300 generic.go:356] "Generic (PLEG): container finished" podID="26f9928d-0276-4c79-800a-9886a99c6499" containerID="e86d4632e9eb84c2bbb3373c31c5d5bce2adaa8155992197eeb4ddb0c13409f1" exitCode=0 Oct 30 01:17:42.677959 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.677922 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" Oct 30 01:17:42.677959 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.677944 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" event={"ID":"26f9928d-0276-4c79-800a-9886a99c6499","Type":"ContainerDied","Data":"e86d4632e9eb84c2bbb3373c31c5d5bce2adaa8155992197eeb4ddb0c13409f1"} Oct 30 01:17:42.678214 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.677992 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk" event={"ID":"26f9928d-0276-4c79-800a-9886a99c6499","Type":"ContainerDied","Data":"acac3317495897580373d4b82410bc2dcae1abbbb1f5b02f7b4bb889dc84a482"} Oct 30 01:17:42.678214 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.678011 5300 scope.go:122] "RemoveContainer" containerID="e86d4632e9eb84c2bbb3373c31c5d5bce2adaa8155992197eeb4ddb0c13409f1" Oct 30 01:17:42.689512 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.689491 5300 scope.go:122] "RemoveContainer" containerID="e86d4632e9eb84c2bbb3373c31c5d5bce2adaa8155992197eeb4ddb0c13409f1" Oct 30 01:17:42.689812 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:17:42.689793 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e86d4632e9eb84c2bbb3373c31c5d5bce2adaa8155992197eeb4ddb0c13409f1\": container with ID starting with e86d4632e9eb84c2bbb3373c31c5d5bce2adaa8155992197eeb4ddb0c13409f1 not found: ID does not exist" containerID="e86d4632e9eb84c2bbb3373c31c5d5bce2adaa8155992197eeb4ddb0c13409f1" Oct 30 01:17:42.689864 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.689822 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e86d4632e9eb84c2bbb3373c31c5d5bce2adaa8155992197eeb4ddb0c13409f1"} err="failed to get container status \"e86d4632e9eb84c2bbb3373c31c5d5bce2adaa8155992197eeb4ddb0c13409f1\": rpc error: code = NotFound desc = could not find container \"e86d4632e9eb84c2bbb3373c31c5d5bce2adaa8155992197eeb4ddb0c13409f1\": container with ID starting with e86d4632e9eb84c2bbb3373c31c5d5bce2adaa8155992197eeb4ddb0c13409f1 not found: ID does not exist" Oct 30 01:17:42.737155 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.737116 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk"] Oct 30 01:17:42.751369 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:42.751331 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e92dab/test-model-e92dab-kserve-686fcb5c77-xcprk"] Oct 30 01:17:43.824067 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:43.824034 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26f9928d-0276-4c79-800a-9886a99c6499" path="/var/lib/kubelet/pods/26f9928d-0276-4c79-800a-9886a99c6499/volumes" Oct 30 01:17:43.938886 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:43.938843 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:45.575244 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:45.575207 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:47.242468 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:47.242437 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:47.301624 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:47.301591 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" Oct 30 01:17:47.322174 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:47.322147 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" Oct 30 01:17:47.727939 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:47.727915 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" Oct 30 01:17:47.901636 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:47.901600 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w"] Oct 30 01:17:48.012842 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:48.012691 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:48.117078 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:48.117040 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:48.746755 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:48.746715 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2"] Oct 30 01:17:48.751742 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:48.751710 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:48.804698 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:48.804661 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9"] Oct 30 01:17:48.914541 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:48.914504 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:49.021340 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:49.021238 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:49.264583 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:49.264547 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" Oct 30 01:17:49.278737 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:49.278656 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" Oct 30 01:17:49.743269 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:49.743234 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" Oct 30 01:17:50.459919 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:50.459869 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:51.595183 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:17:51.595150 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:17:52.351492 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:52.351458 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:52.902240 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:52.902191 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w" podUID="cc41d7ad-a029-4258-9927-b29a5c8e6948" containerName="payload-pre-processing" containerID="cri-o://4c99ed0c88bf2b4a7e2ce8016238dc8720487307833f6b5ebe678392ac1e87c5" gracePeriod=25 Oct 30 01:17:53.141265 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:53.141218 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w" podUID="cc41d7ad-a029-4258-9927-b29a5c8e6948" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:17:53.747851 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:53.747807 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2" podUID="57510f64-4e51-4f32-bfbb-c9730bdb81c3" containerName="payload-processing" containerID="cri-o://70e387913b828fc7779a2ec0809ad3fc2f306621810af2501929bc2024d94489" gracePeriod=25 Oct 30 01:17:53.805860 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:53.805799 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9" podUID="959acd3d-9ea2-49bb-b791-5b87e28d693b" containerName="payload-pre-processing" containerID="cri-o://936a482b38fd7392f484ba736859c1433864dc2d049b14d8742bf9c571f2530e" gracePeriod=25 Oct 30 01:17:53.934561 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:53.934527 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:54.702221 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:54.702188 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:55.643532 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:55.643484 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:56.490919 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:56.490881 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s"] Oct 30 01:17:56.515804 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:56.515775 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:57.404865 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:57.404830 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:17:58.815021 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:58.814984 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9" podUID="959acd3d-9ea2-49bb-b791-5b87e28d693b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:17:58.815021 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:58.814997 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2" podUID="57510f64-4e51-4f32-bfbb-c9730bdb81c3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:17:59.381628 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:17:59.381560 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:01.098132 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:01.097978 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:02.551424 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:02.551386 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:03.141730 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:03.141691 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w" podUID="cc41d7ad-a029-4258-9927-b29a5c8e6948" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:18:04.060895 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:04.060859 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:04.235423 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:04.235382 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:04.431584 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:04.431500 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:04.531994 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:04.531961 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:05.420063 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:05.420030 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:05.803080 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:05.803039 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:07.284128 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:07.284085 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:07.376061 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:07.376024 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:08.815037 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:08.814992 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9" podUID="959acd3d-9ea2-49bb-b791-5b87e28d693b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:18:08.815553 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:08.815091 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2" podUID="57510f64-4e51-4f32-bfbb-c9730bdb81c3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:18:08.864921 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:08.864883 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:10.540577 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:10.540536 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:11.514350 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:11.514302 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" podUID="e84bfe17-6854-4ef1-ae47-288f2dbafb60" containerName="main" containerID="cri-o://2ca56166e386ed2f993690735de91adadd9c1df12a813a57efd3b8e8ae42caa3" gracePeriod=45 Oct 30 01:18:11.814594 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:11.814565 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" Oct 30 01:18:11.919359 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:11.919317 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-tmp-dir\") pod \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\" (UID: \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\") " Oct 30 01:18:11.919660 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:11.919638 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e84bfe17-6854-4ef1-ae47-288f2dbafb60-kube-api-access-zq54m\" (UniqueName: \"kubernetes.io/projected/e84bfe17-6854-4ef1-ae47-288f2dbafb60-kube-api-access-zq54m\") pod \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\" (UID: \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\") " Oct 30 01:18:11.919907 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:11.919651 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-tmp-dir" pod "e84bfe17-6854-4ef1-ae47-288f2dbafb60" (UID: "e84bfe17-6854-4ef1-ae47-288f2dbafb60"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:11.920112 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:11.920020 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-home\" (UniqueName: \"kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-home\") pod \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\" (UID: \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\") " Oct 30 01:18:11.920265 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:11.920242 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-home" pod "e84bfe17-6854-4ef1-ae47-288f2dbafb60" (UID: "e84bfe17-6854-4ef1-ae47-288f2dbafb60"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:11.920579 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:11.920546 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e84bfe17-6854-4ef1-ae47-288f2dbafb60-tls-certs\" (UniqueName: \"kubernetes.io/secret/e84bfe17-6854-4ef1-ae47-288f2dbafb60-tls-certs\") pod \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\" (UID: \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\") " Oct 30 01:18:11.920702 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:11.920607 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-dshm\") pod \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\" (UID: \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\") " Oct 30 01:18:11.920811 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:11.920705 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-model-cache\") pod \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\" (UID: \"e84bfe17-6854-4ef1-ae47-288f2dbafb60\") " Oct 30 01:18:11.921576 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:11.921551 5300 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-home\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:11.921680 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:11.921583 5300 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-tmp-dir\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:11.922595 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:11.922566 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e84bfe17-6854-4ef1-ae47-288f2dbafb60-kube-api-access-zq54m" pod "e84bfe17-6854-4ef1-ae47-288f2dbafb60" (UID: "e84bfe17-6854-4ef1-ae47-288f2dbafb60"). InnerVolumeSpecName "kube-api-access-zq54m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:11.924022 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:11.923996 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-model-cache" pod "e84bfe17-6854-4ef1-ae47-288f2dbafb60" (UID: "e84bfe17-6854-4ef1-ae47-288f2dbafb60"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:11.924540 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:11.924511 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e84bfe17-6854-4ef1-ae47-288f2dbafb60-tls-certs" pod "e84bfe17-6854-4ef1-ae47-288f2dbafb60" (UID: "e84bfe17-6854-4ef1-ae47-288f2dbafb60"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:18:11.925002 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:11.924981 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-dshm" pod "e84bfe17-6854-4ef1-ae47-288f2dbafb60" (UID: "e84bfe17-6854-4ef1-ae47-288f2dbafb60"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:11.941284 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:11.941257 5300 generic.go:356] "Generic (PLEG): container finished" podID="e84bfe17-6854-4ef1-ae47-288f2dbafb60" containerID="2ca56166e386ed2f993690735de91adadd9c1df12a813a57efd3b8e8ae42caa3" exitCode=0 Oct 30 01:18:11.941475 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:11.941300 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" Oct 30 01:18:11.941475 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:11.941348 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" event={"ID":"e84bfe17-6854-4ef1-ae47-288f2dbafb60","Type":"ContainerDied","Data":"2ca56166e386ed2f993690735de91adadd9c1df12a813a57efd3b8e8ae42caa3"} Oct 30 01:18:11.941475 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:11.941392 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" event={"ID":"e84bfe17-6854-4ef1-ae47-288f2dbafb60","Type":"ContainerDied","Data":"92be1bc8de66e976b68a973a0a749bd8b63cfb96d5d4381d6683b9d8b1303cfe"} Oct 30 01:18:11.941475 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:11.941417 5300 scope.go:122] "RemoveContainer" containerID="2ca56166e386ed2f993690735de91adadd9c1df12a813a57efd3b8e8ae42caa3" Oct 30 01:18:11.956010 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:11.955990 5300 scope.go:122] "RemoveContainer" containerID="2ca56166e386ed2f993690735de91adadd9c1df12a813a57efd3b8e8ae42caa3" Oct 30 01:18:11.956313 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:18:11.956294 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ca56166e386ed2f993690735de91adadd9c1df12a813a57efd3b8e8ae42caa3\": container with ID starting with 2ca56166e386ed2f993690735de91adadd9c1df12a813a57efd3b8e8ae42caa3 not found: ID does not exist" containerID="2ca56166e386ed2f993690735de91adadd9c1df12a813a57efd3b8e8ae42caa3" Oct 30 01:18:11.956358 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:11.956322 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ca56166e386ed2f993690735de91adadd9c1df12a813a57efd3b8e8ae42caa3"} err="failed to get container status \"2ca56166e386ed2f993690735de91adadd9c1df12a813a57efd3b8e8ae42caa3\": rpc error: code = NotFound desc = could not find container \"2ca56166e386ed2f993690735de91adadd9c1df12a813a57efd3b8e8ae42caa3\": container with ID starting with 2ca56166e386ed2f993690735de91adadd9c1df12a813a57efd3b8e8ae42caa3 not found: ID does not exist" Oct 30 01:18:11.999248 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:11.999213 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s"] Oct 30 01:18:12.003838 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:12.003808 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s"] Oct 30 01:18:12.023590 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:12.023553 5300 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e84bfe17-6854-4ef1-ae47-288f2dbafb60-tls-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:12.023590 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:12.023583 5300 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-dshm\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:12.023590 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:12.023592 5300 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e84bfe17-6854-4ef1-ae47-288f2dbafb60-model-cache\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:12.023831 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:12.023600 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq54m\" (UniqueName: \"kubernetes.io/projected/e84bfe17-6854-4ef1-ae47-288f2dbafb60-kube-api-access-zq54m\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:12.218298 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:12.218211 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:12.755298 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:12.755261 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-56d1d9/test-model-56d1d9-kserve-65d4df9bdf-7n97s" podUID="e84bfe17-6854-4ef1-ae47-288f2dbafb60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.122:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:18:13.189471 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:13.189442 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w" Oct 30 01:18:13.344047 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:13.343956 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc41d7ad-a029-4258-9927-b29a5c8e6948-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc41d7ad-a029-4258-9927-b29a5c8e6948-plugins-config-volume\") pod \"cc41d7ad-a029-4258-9927-b29a5c8e6948\" (UID: \"cc41d7ad-a029-4258-9927-b29a5c8e6948\") " Oct 30 01:18:13.344047 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:13.344006 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc41d7ad-a029-4258-9927-b29a5c8e6948-kube-api-access-b5mnv\" (UniqueName: \"kubernetes.io/projected/cc41d7ad-a029-4258-9927-b29a5c8e6948-kube-api-access-b5mnv\") pod \"cc41d7ad-a029-4258-9927-b29a5c8e6948\" (UID: \"cc41d7ad-a029-4258-9927-b29a5c8e6948\") " Oct 30 01:18:13.344354 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:13.344328 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc41d7ad-a029-4258-9927-b29a5c8e6948-plugins-config-volume" pod "cc41d7ad-a029-4258-9927-b29a5c8e6948" (UID: "cc41d7ad-a029-4258-9927-b29a5c8e6948"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:13.344596 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:13.344581 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc41d7ad-a029-4258-9927-b29a5c8e6948-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:13.346236 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:13.346212 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc41d7ad-a029-4258-9927-b29a5c8e6948-kube-api-access-b5mnv" pod "cc41d7ad-a029-4258-9927-b29a5c8e6948" (UID: "cc41d7ad-a029-4258-9927-b29a5c8e6948"). InnerVolumeSpecName "kube-api-access-b5mnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:13.446284 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:13.446247 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5mnv\" (UniqueName: \"kubernetes.io/projected/cc41d7ad-a029-4258-9927-b29a5c8e6948-kube-api-access-b5mnv\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:13.817278 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:13.817240 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:13.824487 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:13.824453 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e84bfe17-6854-4ef1-ae47-288f2dbafb60" path="/var/lib/kubelet/pods/e84bfe17-6854-4ef1-ae47-288f2dbafb60/volumes" Oct 30 01:18:13.983484 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:13.983268 5300 generic.go:356] "Generic (PLEG): container finished" podID="57510f64-4e51-4f32-bfbb-c9730bdb81c3" containerID="70e387913b828fc7779a2ec0809ad3fc2f306621810af2501929bc2024d94489" exitCode=0 Oct 30 01:18:13.983654 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:13.983605 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2" event={"ID":"57510f64-4e51-4f32-bfbb-c9730bdb81c3","Type":"ContainerDied","Data":"70e387913b828fc7779a2ec0809ad3fc2f306621810af2501929bc2024d94489"} Oct 30 01:18:13.990849 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:13.986822 5300 generic.go:356] "Generic (PLEG): container finished" podID="959acd3d-9ea2-49bb-b791-5b87e28d693b" containerID="936a482b38fd7392f484ba736859c1433864dc2d049b14d8742bf9c571f2530e" exitCode=0 Oct 30 01:18:13.990849 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:13.986954 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9" event={"ID":"959acd3d-9ea2-49bb-b791-5b87e28d693b","Type":"ContainerDied","Data":"936a482b38fd7392f484ba736859c1433864dc2d049b14d8742bf9c571f2530e"} Oct 30 01:18:13.990849 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:13.990017 5300 generic.go:356] "Generic (PLEG): container finished" podID="cc41d7ad-a029-4258-9927-b29a5c8e6948" containerID="4c99ed0c88bf2b4a7e2ce8016238dc8720487307833f6b5ebe678392ac1e87c5" exitCode=0 Oct 30 01:18:13.990849 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:13.990110 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w" event={"ID":"cc41d7ad-a029-4258-9927-b29a5c8e6948","Type":"ContainerDied","Data":"4c99ed0c88bf2b4a7e2ce8016238dc8720487307833f6b5ebe678392ac1e87c5"} Oct 30 01:18:13.990849 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:13.990156 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w" event={"ID":"cc41d7ad-a029-4258-9927-b29a5c8e6948","Type":"ContainerDied","Data":"2fd69213b9d5b4c3f27f6fb2ee924324c191fff4f34ffeea7229c342fe2cddd0"} Oct 30 01:18:13.990849 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:13.990198 5300 scope.go:122] "RemoveContainer" containerID="4c99ed0c88bf2b4a7e2ce8016238dc8720487307833f6b5ebe678392ac1e87c5" Oct 30 01:18:13.990849 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:13.990309 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w" Oct 30 01:18:14.010195 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:14.008909 5300 scope.go:122] "RemoveContainer" containerID="4c99ed0c88bf2b4a7e2ce8016238dc8720487307833f6b5ebe678392ac1e87c5" Oct 30 01:18:14.011030 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:18:14.010883 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c99ed0c88bf2b4a7e2ce8016238dc8720487307833f6b5ebe678392ac1e87c5\": container with ID starting with 4c99ed0c88bf2b4a7e2ce8016238dc8720487307833f6b5ebe678392ac1e87c5 not found: ID does not exist" containerID="4c99ed0c88bf2b4a7e2ce8016238dc8720487307833f6b5ebe678392ac1e87c5" Oct 30 01:18:14.011030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:14.010960 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c99ed0c88bf2b4a7e2ce8016238dc8720487307833f6b5ebe678392ac1e87c5"} err="failed to get container status \"4c99ed0c88bf2b4a7e2ce8016238dc8720487307833f6b5ebe678392ac1e87c5\": rpc error: code = NotFound desc = could not find container \"4c99ed0c88bf2b4a7e2ce8016238dc8720487307833f6b5ebe678392ac1e87c5\": container with ID starting with 4c99ed0c88bf2b4a7e2ce8016238dc8720487307833f6b5ebe678392ac1e87c5 not found: ID does not exist" Oct 30 01:18:14.057548 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:14.057517 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w"] Oct 30 01:18:14.073460 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:14.073422 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w"] Oct 30 01:18:14.133459 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:14.133428 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2" Oct 30 01:18:14.140712 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:14.140670 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-687471-57fb85bd67-twh2w" podUID="cc41d7ad-a029-4258-9927-b29a5c8e6948" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.113:50052\" within 1s: context deadline exceeded" Oct 30 01:18:14.140910 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:18:14.140791 5300 logging.go:55] [core] [Channel #4868 SubChannel #4869]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.113:50052", ServerName: "10.133.0.113:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.113:50052: operation was canceled" Oct 30 01:18:14.198265 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:14.198231 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9" Oct 30 01:18:14.258421 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:14.258337 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57510f64-4e51-4f32-bfbb-c9730bdb81c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57510f64-4e51-4f32-bfbb-c9730bdb81c3-plugins-config-volume\") pod \"57510f64-4e51-4f32-bfbb-c9730bdb81c3\" (UID: \"57510f64-4e51-4f32-bfbb-c9730bdb81c3\") " Oct 30 01:18:14.258598 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:14.258438 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57510f64-4e51-4f32-bfbb-c9730bdb81c3-kube-api-access-bntmh\" (UniqueName: \"kubernetes.io/projected/57510f64-4e51-4f32-bfbb-c9730bdb81c3-kube-api-access-bntmh\") pod \"57510f64-4e51-4f32-bfbb-c9730bdb81c3\" (UID: \"57510f64-4e51-4f32-bfbb-c9730bdb81c3\") " Oct 30 01:18:14.258889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:14.258858 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57510f64-4e51-4f32-bfbb-c9730bdb81c3-plugins-config-volume" pod "57510f64-4e51-4f32-bfbb-c9730bdb81c3" (UID: "57510f64-4e51-4f32-bfbb-c9730bdb81c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:14.259058 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:14.259033 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57510f64-4e51-4f32-bfbb-c9730bdb81c3-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:14.260774 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:14.260737 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57510f64-4e51-4f32-bfbb-c9730bdb81c3-kube-api-access-bntmh" pod "57510f64-4e51-4f32-bfbb-c9730bdb81c3" (UID: "57510f64-4e51-4f32-bfbb-c9730bdb81c3"). InnerVolumeSpecName "kube-api-access-bntmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:14.359624 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:14.359585 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/959acd3d-9ea2-49bb-b791-5b87e28d693b-kube-api-access-7v875\" (UniqueName: \"kubernetes.io/projected/959acd3d-9ea2-49bb-b791-5b87e28d693b-kube-api-access-7v875\") pod \"959acd3d-9ea2-49bb-b791-5b87e28d693b\" (UID: \"959acd3d-9ea2-49bb-b791-5b87e28d693b\") " Oct 30 01:18:14.359870 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:14.359852 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/959acd3d-9ea2-49bb-b791-5b87e28d693b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959acd3d-9ea2-49bb-b791-5b87e28d693b-plugins-config-volume\") pod \"959acd3d-9ea2-49bb-b791-5b87e28d693b\" (UID: \"959acd3d-9ea2-49bb-b791-5b87e28d693b\") " Oct 30 01:18:14.360196 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:14.360172 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/959acd3d-9ea2-49bb-b791-5b87e28d693b-plugins-config-volume" pod "959acd3d-9ea2-49bb-b791-5b87e28d693b" (UID: "959acd3d-9ea2-49bb-b791-5b87e28d693b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:14.360401 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:14.360384 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959acd3d-9ea2-49bb-b791-5b87e28d693b-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:14.360460 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:14.360407 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bntmh\" (UniqueName: \"kubernetes.io/projected/57510f64-4e51-4f32-bfbb-c9730bdb81c3-kube-api-access-bntmh\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:14.361942 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:14.361920 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/959acd3d-9ea2-49bb-b791-5b87e28d693b-kube-api-access-7v875" pod "959acd3d-9ea2-49bb-b791-5b87e28d693b" (UID: "959acd3d-9ea2-49bb-b791-5b87e28d693b"). InnerVolumeSpecName "kube-api-access-7v875". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:14.462320 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:14.462275 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v875\" (UniqueName: \"kubernetes.io/projected/959acd3d-9ea2-49bb-b791-5b87e28d693b-kube-api-access-7v875\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:14.998798 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:14.998752 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2" Oct 30 01:18:14.998798 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:14.998792 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2" event={"ID":"57510f64-4e51-4f32-bfbb-c9730bdb81c3","Type":"ContainerDied","Data":"52e1d85707d235ef4513ee015f958e0889fc2fcf8712c789cce96ce63a67074e"} Oct 30 01:18:14.999024 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:14.998848 5300 scope.go:122] "RemoveContainer" containerID="70e387913b828fc7779a2ec0809ad3fc2f306621810af2501929bc2024d94489" Oct 30 01:18:15.000408 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:15.000388 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9" Oct 30 01:18:15.000528 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:15.000411 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9" event={"ID":"959acd3d-9ea2-49bb-b791-5b87e28d693b","Type":"ContainerDied","Data":"e71754692a8d2e921e65f6b4b754ebbc7623bb12d83a060b0c884434190f54b7"} Oct 30 01:18:15.010602 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:15.010553 5300 scope.go:122] "RemoveContainer" containerID="936a482b38fd7392f484ba736859c1433864dc2d049b14d8742bf9c571f2530e" Oct 30 01:18:15.056266 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:15.056229 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9"] Oct 30 01:18:15.069821 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:15.069790 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4dde48-745c9456d7-k5tq9"] Oct 30 01:18:15.100142 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:15.100109 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2"] Oct 30 01:18:15.103731 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:15.103709 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4dde48-8f44b4878-66jr2"] Oct 30 01:18:15.450571 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:15.450539 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:15.824274 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:15.824241 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57510f64-4e51-4f32-bfbb-c9730bdb81c3" path="/var/lib/kubelet/pods/57510f64-4e51-4f32-bfbb-c9730bdb81c3/volumes" Oct 30 01:18:15.824677 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:15.824663 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="959acd3d-9ea2-49bb-b791-5b87e28d693b" path="/var/lib/kubelet/pods/959acd3d-9ea2-49bb-b791-5b87e28d693b/volumes" Oct 30 01:18:15.825183 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:15.825165 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc41d7ad-a029-4258-9927-b29a5c8e6948" path="/var/lib/kubelet/pods/cc41d7ad-a029-4258-9927-b29a5c8e6948/volumes" Oct 30 01:18:17.020091 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:17.020051 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:18.459630 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:18.459597 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq"] Oct 30 01:18:18.460502 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:18.460459 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq" podUID="00621ae7-7e53-415c-a3a7-174a79440f4d" containerName="maas-api" containerID="cri-o://3c647ceee31f932063cce742e9206ca57283ef77e890c66367dafd1bea13ee55" gracePeriod=30 Oct 30 01:18:18.505038 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:18.504963 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:18.781344 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:18.781310 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq" Oct 30 01:18:18.837533 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:18.836247 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00621ae7-7e53-415c-a3a7-174a79440f4d-kube-api-access-7rr6r\" (UniqueName: \"kubernetes.io/projected/00621ae7-7e53-415c-a3a7-174a79440f4d-kube-api-access-7rr6r\") pod \"00621ae7-7e53-415c-a3a7-174a79440f4d\" (UID: \"00621ae7-7e53-415c-a3a7-174a79440f4d\") " Oct 30 01:18:18.837533 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:18.836303 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00621ae7-7e53-415c-a3a7-174a79440f4d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00621ae7-7e53-415c-a3a7-174a79440f4d-openshift-service-ca\") pod \"00621ae7-7e53-415c-a3a7-174a79440f4d\" (UID: \"00621ae7-7e53-415c-a3a7-174a79440f4d\") " Oct 30 01:18:18.837533 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:18.836343 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00621ae7-7e53-415c-a3a7-174a79440f4d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/00621ae7-7e53-415c-a3a7-174a79440f4d-metrics-certs\") pod \"00621ae7-7e53-415c-a3a7-174a79440f4d\" (UID: \"00621ae7-7e53-415c-a3a7-174a79440f4d\") " Oct 30 01:18:18.837533 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:18.836488 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00621ae7-7e53-415c-a3a7-174a79440f4d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00621ae7-7e53-415c-a3a7-174a79440f4d-maas-api-tls\") pod \"00621ae7-7e53-415c-a3a7-174a79440f4d\" (UID: \"00621ae7-7e53-415c-a3a7-174a79440f4d\") " Oct 30 01:18:18.837533 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:18.837205 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00621ae7-7e53-415c-a3a7-174a79440f4d-openshift-service-ca" pod "00621ae7-7e53-415c-a3a7-174a79440f4d" (UID: "00621ae7-7e53-415c-a3a7-174a79440f4d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:18.853005 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:18.848960 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00621ae7-7e53-415c-a3a7-174a79440f4d-metrics-certs" pod "00621ae7-7e53-415c-a3a7-174a79440f4d" (UID: "00621ae7-7e53-415c-a3a7-174a79440f4d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:18:18.853005 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:18.849018 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00621ae7-7e53-415c-a3a7-174a79440f4d-kube-api-access-7rr6r" pod "00621ae7-7e53-415c-a3a7-174a79440f4d" (UID: "00621ae7-7e53-415c-a3a7-174a79440f4d"). InnerVolumeSpecName "kube-api-access-7rr6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:18.853005 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:18.849137 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00621ae7-7e53-415c-a3a7-174a79440f4d-maas-api-tls" pod "00621ae7-7e53-415c-a3a7-174a79440f4d" (UID: "00621ae7-7e53-415c-a3a7-174a79440f4d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:18:18.938665 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:18.938630 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00621ae7-7e53-415c-a3a7-174a79440f4d-maas-api-tls\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:18.938665 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:18.938664 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rr6r\" (UniqueName: \"kubernetes.io/projected/00621ae7-7e53-415c-a3a7-174a79440f4d-kube-api-access-7rr6r\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:18.938665 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:18.938675 5300 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00621ae7-7e53-415c-a3a7-174a79440f4d-openshift-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:18.938933 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:18.938685 5300 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00621ae7-7e53-415c-a3a7-174a79440f4d-metrics-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:18.972127 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:18.972088 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:19.040622 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:19.040523 5300 generic.go:356] "Generic (PLEG): container finished" podID="00621ae7-7e53-415c-a3a7-174a79440f4d" containerID="3c647ceee31f932063cce742e9206ca57283ef77e890c66367dafd1bea13ee55" exitCode=0 Oct 30 01:18:19.040622 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:19.040571 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq" Oct 30 01:18:19.040622 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:19.040605 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq" event={"ID":"00621ae7-7e53-415c-a3a7-174a79440f4d","Type":"ContainerDied","Data":"3c647ceee31f932063cce742e9206ca57283ef77e890c66367dafd1bea13ee55"} Oct 30 01:18:19.040887 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:19.040664 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq" event={"ID":"00621ae7-7e53-415c-a3a7-174a79440f4d","Type":"ContainerDied","Data":"41e02190b5aee359639a746b52c8be38fea85b5a9ce9f709f752d23360af1f90"} Oct 30 01:18:19.040887 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:19.040693 5300 scope.go:122] "RemoveContainer" containerID="3c647ceee31f932063cce742e9206ca57283ef77e890c66367dafd1bea13ee55" Oct 30 01:18:19.057554 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:19.056471 5300 scope.go:122] "RemoveContainer" containerID="3c647ceee31f932063cce742e9206ca57283ef77e890c66367dafd1bea13ee55" Oct 30 01:18:19.062563 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:18:19.062527 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c647ceee31f932063cce742e9206ca57283ef77e890c66367dafd1bea13ee55\": container with ID starting with 3c647ceee31f932063cce742e9206ca57283ef77e890c66367dafd1bea13ee55 not found: ID does not exist" containerID="3c647ceee31f932063cce742e9206ca57283ef77e890c66367dafd1bea13ee55" Oct 30 01:18:19.062688 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:19.062573 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c647ceee31f932063cce742e9206ca57283ef77e890c66367dafd1bea13ee55"} err="failed to get container status \"3c647ceee31f932063cce742e9206ca57283ef77e890c66367dafd1bea13ee55\": rpc error: code = NotFound desc = could not find container \"3c647ceee31f932063cce742e9206ca57283ef77e890c66367dafd1bea13ee55\": container with ID starting with 3c647ceee31f932063cce742e9206ca57283ef77e890c66367dafd1bea13ee55 not found: ID does not exist" Oct 30 01:18:19.090991 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:19.090956 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:19.102685 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:19.102648 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq"] Oct 30 01:18:19.106422 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:19.106394 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e92dab-7487d8b95d-ck9zq"] Oct 30 01:18:19.204453 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:19.204417 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:19.588731 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:19.588694 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:19.826819 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:19.826786 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00621ae7-7e53-415c-a3a7-174a79440f4d" path="/var/lib/kubelet/pods/00621ae7-7e53-415c-a3a7-174a79440f4d/volumes" Oct 30 01:18:20.603771 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:20.603721 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:21.100193 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:21.100159 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:21.216336 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:21.216296 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:21.428898 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:21.428807 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:21.535951 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:21.535916 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:21.599577 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:18:21.599540 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:18:21.749079 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:21.749039 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:23.131025 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:23.130993 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:24.451386 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:24.451346 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs"] Oct 30 01:18:24.451923 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:24.451605 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs" podUID="2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9" containerName="maas-api" containerID="cri-o://1c408594fa159605ba6043627729d80c0f1ba5aea5d22c51257b7f6b31ccda5d" gracePeriod=30 Oct 30 01:18:24.507750 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:24.507714 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:24.763010 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:24.762978 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs" Oct 30 01:18:24.863154 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:24.863098 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-maas-api-tls\") pod \"2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9\" (UID: \"2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9\") " Oct 30 01:18:24.863370 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:24.863173 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-kube-api-access-v7mcd\" (UniqueName: \"kubernetes.io/projected/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-kube-api-access-v7mcd\") pod \"2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9\" (UID: \"2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9\") " Oct 30 01:18:24.863370 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:24.863351 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-metrics-certs\") pod \"2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9\" (UID: \"2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9\") " Oct 30 01:18:24.863783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:24.863517 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-openshift-service-ca\") pod \"2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9\" (UID: \"2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9\") " Oct 30 01:18:24.879374 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:24.878399 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-metrics-certs" pod "2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9" (UID: "2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:18:24.879374 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:24.878798 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-kube-api-access-v7mcd" pod "2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9" (UID: "2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9"). InnerVolumeSpecName "kube-api-access-v7mcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:24.879374 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:24.878990 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-maas-api-tls" pod "2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9" (UID: "2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:18:24.896073 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:24.896025 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-openshift-service-ca" pod "2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9" (UID: "2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:24.968878 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:24.968013 5300 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-metrics-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:24.969111 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:24.969088 5300 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-openshift-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:24.969195 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:24.969184 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-maas-api-tls\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:24.969286 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:24.969274 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7mcd\" (UniqueName: \"kubernetes.io/projected/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9-kube-api-access-v7mcd\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:24.991072 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:24.991029 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:25.090670 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.090635 5300 generic.go:356] "Generic (PLEG): container finished" podID="2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9" containerID="1c408594fa159605ba6043627729d80c0f1ba5aea5d22c51257b7f6b31ccda5d" exitCode=0 Oct 30 01:18:25.090906 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.090706 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs" Oct 30 01:18:25.090906 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.090716 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs" event={"ID":"2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9","Type":"ContainerDied","Data":"1c408594fa159605ba6043627729d80c0f1ba5aea5d22c51257b7f6b31ccda5d"} Oct 30 01:18:25.090906 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.090853 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs" event={"ID":"2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9","Type":"ContainerDied","Data":"dac2dc82a55958258371d6596dcc6375babf2c27c6a5a96d1d21c33ce5dde409"} Oct 30 01:18:25.090906 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.090887 5300 scope.go:122] "RemoveContainer" containerID="1c408594fa159605ba6043627729d80c0f1ba5aea5d22c51257b7f6b31ccda5d" Oct 30 01:18:25.101861 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.101842 5300 scope.go:122] "RemoveContainer" containerID="1c408594fa159605ba6043627729d80c0f1ba5aea5d22c51257b7f6b31ccda5d" Oct 30 01:18:25.102319 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:18:25.102293 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c408594fa159605ba6043627729d80c0f1ba5aea5d22c51257b7f6b31ccda5d\": container with ID starting with 1c408594fa159605ba6043627729d80c0f1ba5aea5d22c51257b7f6b31ccda5d not found: ID does not exist" containerID="1c408594fa159605ba6043627729d80c0f1ba5aea5d22c51257b7f6b31ccda5d" Oct 30 01:18:25.102434 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.102325 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c408594fa159605ba6043627729d80c0f1ba5aea5d22c51257b7f6b31ccda5d"} err="failed to get container status \"1c408594fa159605ba6043627729d80c0f1ba5aea5d22c51257b7f6b31ccda5d\": rpc error: code = NotFound desc = could not find container \"1c408594fa159605ba6043627729d80c0f1ba5aea5d22c51257b7f6b31ccda5d\": container with ID starting with 1c408594fa159605ba6043627729d80c0f1ba5aea5d22c51257b7f6b31ccda5d not found: ID does not exist" Oct 30 01:18:25.113422 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.113380 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:25.151666 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.151632 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs"] Oct 30 01:18:25.161742 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.161704 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4dde48-75b74d578-k2pcs"] Oct 30 01:18:25.208321 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.208284 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:25.579808 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.579772 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:25.689135 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.689097 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:25.772976 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.772939 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j"] Oct 30 01:18:25.773593 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.773580 5300 cpu_manager.go:414] "containerMap: removing container" podUID="cc41d7ad-a029-4258-9927-b29a5c8e6948" containerName="payload-pre-processing" Oct 30 01:18:25.773635 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.773595 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc41d7ad-a029-4258-9927-b29a5c8e6948" containerName="payload-pre-processing" Oct 30 01:18:25.773635 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.773612 5300 cpu_manager.go:414] "containerMap: removing container" podUID="57510f64-4e51-4f32-bfbb-c9730bdb81c3" containerName="payload-processing" Oct 30 01:18:25.773635 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.773617 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57510f64-4e51-4f32-bfbb-c9730bdb81c3" containerName="payload-processing" Oct 30 01:18:25.773780 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.773641 5300 cpu_manager.go:414] "containerMap: removing container" podUID="26f9928d-0276-4c79-800a-9886a99c6499" containerName="main" Oct 30 01:18:25.773780 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.773646 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f9928d-0276-4c79-800a-9886a99c6499" containerName="main" Oct 30 01:18:25.773780 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.773657 5300 cpu_manager.go:414] "containerMap: removing container" podUID="e84bfe17-6854-4ef1-ae47-288f2dbafb60" containerName="main" Oct 30 01:18:25.773780 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.773663 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e84bfe17-6854-4ef1-ae47-288f2dbafb60" containerName="main" Oct 30 01:18:25.773780 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.773673 5300 cpu_manager.go:414] "containerMap: removing container" podUID="959acd3d-9ea2-49bb-b791-5b87e28d693b" containerName="payload-pre-processing" Oct 30 01:18:25.773780 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.773677 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="959acd3d-9ea2-49bb-b791-5b87e28d693b" containerName="payload-pre-processing" Oct 30 01:18:25.773780 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.773687 5300 cpu_manager.go:414] "containerMap: removing container" podUID="2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9" containerName="maas-api" Oct 30 01:18:25.773780 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.773692 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9" containerName="maas-api" Oct 30 01:18:25.773780 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.773699 5300 cpu_manager.go:414] "containerMap: removing container" podUID="00621ae7-7e53-415c-a3a7-174a79440f4d" containerName="maas-api" Oct 30 01:18:25.773780 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.773703 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00621ae7-7e53-415c-a3a7-174a79440f4d" containerName="maas-api" Oct 30 01:18:25.774095 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.773815 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc41d7ad-a029-4258-9927-b29a5c8e6948" containerName="payload-pre-processing" Oct 30 01:18:25.774095 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.773826 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="57510f64-4e51-4f32-bfbb-c9730bdb81c3" containerName="payload-processing" Oct 30 01:18:25.774095 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.773836 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f9928d-0276-4c79-800a-9886a99c6499" containerName="main" Oct 30 01:18:25.774095 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.773842 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="e84bfe17-6854-4ef1-ae47-288f2dbafb60" containerName="main" Oct 30 01:18:25.774095 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.773849 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="959acd3d-9ea2-49bb-b791-5b87e28d693b" containerName="payload-pre-processing" Oct 30 01:18:25.774095 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.773856 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9" containerName="maas-api" Oct 30 01:18:25.774095 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.773862 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="00621ae7-7e53-415c-a3a7-174a79440f4d" containerName="maas-api" Oct 30 01:18:25.778508 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.778485 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" Oct 30 01:18:25.798596 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.798562 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j"] Oct 30 01:18:25.806820 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.806439 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-29145c\"/\"openshift-service-ca.crt\"" Oct 30 01:18:25.806820 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.806572 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-29145c\"/\"kube-root-ca.crt\"" Oct 30 01:18:25.826256 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.826216 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9" path="/var/lib/kubelet/pods/2fd50e96-1b82-4e81-9c21-dc35bd0e3ff9/volumes" Oct 30 01:18:25.835300 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.835213 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:25.869823 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.869743 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-29145c\"/\"default-dockercfg-qr8vs\"" Oct 30 01:18:25.870054 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.869992 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-29145c\"/\"sub-test-model-29145c-kserve-self-signed-certs\"" Oct 30 01:18:25.884101 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.884068 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-tmp-dir\") pod \"sub-test-model-29145c-kserve-8444cf8dbf-8xq4j\" (UID: \"31ba3725-cc95-463d-9580-513e3a0354c0\") " pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" Oct 30 01:18:25.884293 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.884115 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-home\") pod \"sub-test-model-29145c-kserve-8444cf8dbf-8xq4j\" (UID: \"31ba3725-cc95-463d-9580-513e3a0354c0\") " pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" Oct 30 01:18:25.884293 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.884239 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwdw4\" (UniqueName: \"kubernetes.io/projected/31ba3725-cc95-463d-9580-513e3a0354c0-kube-api-access-hwdw4\") pod \"sub-test-model-29145c-kserve-8444cf8dbf-8xq4j\" (UID: \"31ba3725-cc95-463d-9580-513e3a0354c0\") " pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" Oct 30 01:18:25.884486 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.884335 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ba3725-cc95-463d-9580-513e3a0354c0-tls-certs\") pod \"sub-test-model-29145c-kserve-8444cf8dbf-8xq4j\" (UID: \"31ba3725-cc95-463d-9580-513e3a0354c0\") " pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" Oct 30 01:18:25.884486 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.884390 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-dshm\") pod \"sub-test-model-29145c-kserve-8444cf8dbf-8xq4j\" (UID: \"31ba3725-cc95-463d-9580-513e3a0354c0\") " pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" Oct 30 01:18:25.884486 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.884457 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-model-cache\") pod \"sub-test-model-29145c-kserve-8444cf8dbf-8xq4j\" (UID: \"31ba3725-cc95-463d-9580-513e3a0354c0\") " pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" Oct 30 01:18:25.959343 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.959309 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:25.985564 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.985529 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-tmp-dir\") pod \"sub-test-model-29145c-kserve-8444cf8dbf-8xq4j\" (UID: \"31ba3725-cc95-463d-9580-513e3a0354c0\") " pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" Oct 30 01:18:25.985917 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.985576 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-home\") pod \"sub-test-model-29145c-kserve-8444cf8dbf-8xq4j\" (UID: \"31ba3725-cc95-463d-9580-513e3a0354c0\") " pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" Oct 30 01:18:25.985917 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.985644 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwdw4\" (UniqueName: \"kubernetes.io/projected/31ba3725-cc95-463d-9580-513e3a0354c0-kube-api-access-hwdw4\") pod \"sub-test-model-29145c-kserve-8444cf8dbf-8xq4j\" (UID: \"31ba3725-cc95-463d-9580-513e3a0354c0\") " pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" Oct 30 01:18:25.985917 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.985732 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ba3725-cc95-463d-9580-513e3a0354c0-tls-certs\") pod \"sub-test-model-29145c-kserve-8444cf8dbf-8xq4j\" (UID: \"31ba3725-cc95-463d-9580-513e3a0354c0\") " pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" Oct 30 01:18:25.985917 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.985803 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-dshm\") pod \"sub-test-model-29145c-kserve-8444cf8dbf-8xq4j\" (UID: \"31ba3725-cc95-463d-9580-513e3a0354c0\") " pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" Oct 30 01:18:25.985917 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.985849 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-model-cache\") pod \"sub-test-model-29145c-kserve-8444cf8dbf-8xq4j\" (UID: \"31ba3725-cc95-463d-9580-513e3a0354c0\") " pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" Oct 30 01:18:25.987286 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.987259 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-model-cache\") pod \"sub-test-model-29145c-kserve-8444cf8dbf-8xq4j\" (UID: \"31ba3725-cc95-463d-9580-513e3a0354c0\") " pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" Oct 30 01:18:25.987286 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.987478 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-tmp-dir\") pod \"sub-test-model-29145c-kserve-8444cf8dbf-8xq4j\" (UID: \"31ba3725-cc95-463d-9580-513e3a0354c0\") " pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" Oct 30 01:18:25.987833 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.987739 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-home\") pod \"sub-test-model-29145c-kserve-8444cf8dbf-8xq4j\" (UID: \"31ba3725-cc95-463d-9580-513e3a0354c0\") " pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" Oct 30 01:18:25.992071 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.992049 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-dshm\") pod \"sub-test-model-29145c-kserve-8444cf8dbf-8xq4j\" (UID: \"31ba3725-cc95-463d-9580-513e3a0354c0\") " pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" Oct 30 01:18:25.992458 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:25.992435 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ba3725-cc95-463d-9580-513e3a0354c0-tls-certs\") pod \"sub-test-model-29145c-kserve-8444cf8dbf-8xq4j\" (UID: \"31ba3725-cc95-463d-9580-513e3a0354c0\") " pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" Oct 30 01:18:26.008967 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:26.008931 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwdw4\" (UniqueName: \"kubernetes.io/projected/31ba3725-cc95-463d-9580-513e3a0354c0-kube-api-access-hwdw4\") pod \"sub-test-model-29145c-kserve-8444cf8dbf-8xq4j\" (UID: \"31ba3725-cc95-463d-9580-513e3a0354c0\") " pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" Oct 30 01:18:26.093886 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:26.093019 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" Oct 30 01:18:26.477515 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:26.477480 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j"] Oct 30 01:18:26.480329 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:18:26.480295 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31ba3725_cc95_463d_9580_513e3a0354c0.slice/crio-6bd699c1a86815bdc89a56bc49c51654f309e41b687c2c7866934183f4d33b1b WatchSource:0}: Error finding container 6bd699c1a86815bdc89a56bc49c51654f309e41b687c2c7866934183f4d33b1b: Status 404 returned error can't find the container with id 6bd699c1a86815bdc89a56bc49c51654f309e41b687c2c7866934183f4d33b1b Oct 30 01:18:27.115407 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:27.115372 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" event={"ID":"31ba3725-cc95-463d-9580-513e3a0354c0","Type":"ContainerStarted","Data":"72ba100c0106b7f72b67645e16d273975b8a0efcd482dd78f2a35a0b448580c7"} Oct 30 01:18:27.115407 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:27.115415 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" event={"ID":"31ba3725-cc95-463d-9580-513e3a0354c0","Type":"ContainerStarted","Data":"6bd699c1a86815bdc89a56bc49c51654f309e41b687c2c7866934183f4d33b1b"} Oct 30 01:18:27.162949 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:27.162886 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" podStartSLOduration=2.162870851 podStartE2EDuration="2.162870851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:27.159638725 +0000 UTC m=+2107.475507063" watchObservedRunningTime="2026-10-30 01:18:27.162870851 +0000 UTC m=+2107.478739175" Oct 30 01:18:27.248804 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:27.248739 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:27.675626 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:27.675588 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:28.024843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:28.024799 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:28.093728 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:28.093688 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" Oct 30 01:18:28.980362 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:28.980330 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:29.091059 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.091018 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:29.188953 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.188911 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:29.390668 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.390580 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:29.781509 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.781474 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq"] Oct 30 01:18:29.786264 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.786242 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:29.835214 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.835180 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq"] Oct 30 01:18:29.861888 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.861847 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-podinfo\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:29.862057 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.861897 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-workload-socket\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:29.862057 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.862010 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-envoy\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:29.862057 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.862054 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-data\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:29.862210 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.862188 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-workload-certs\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:29.862252 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.862238 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njzns\" (UniqueName: \"kubernetes.io/projected/2a68439c-dee6-4fb4-857d-bd09bbe8e214-kube-api-access-njzns\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:29.862389 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.862365 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-token\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:29.862467 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.862447 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-credential-socket\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:29.862499 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.862480 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istiod-ca-cert\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:29.881291 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.881259 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-eacea3-openshift-default-dockercfg-bqqn7\"" Oct 30 01:18:29.964003 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.963961 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-workload-certs\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:29.964003 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.964008 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njzns\" (UniqueName: \"kubernetes.io/projected/2a68439c-dee6-4fb4-857d-bd09bbe8e214-kube-api-access-njzns\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:29.964278 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.964124 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-token\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:29.964278 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.964169 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-credential-socket\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:29.964278 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.964195 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istiod-ca-cert\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:29.964444 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.964373 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-podinfo\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:29.964444 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.964411 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-workload-socket\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:29.964549 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.964445 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-workload-certs\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:29.964606 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.964552 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-envoy\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:29.964656 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.964578 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-credential-socket\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:29.964656 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.964629 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-data\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:29.964989 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.964710 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-workload-socket\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:29.964989 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.964963 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istiod-ca-cert\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:29.965121 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.965089 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-data\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:29.966863 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.966835 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-envoy\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:29.967085 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.967066 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-podinfo\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:29.985185 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.985158 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-token\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:29.985586 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:29.985304 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njzns\" (UniqueName: \"kubernetes.io/projected/2a68439c-dee6-4fb4-857d-bd09bbe8e214-kube-api-access-njzns\") pod \"e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:30.101917 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.101814 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:30.105069 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.105045 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm"] Oct 30 01:18:30.105257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.105235 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" podUID="7d744ad6-75be-4a86-be0f-946122f5302e" containerName="discovery" containerID="cri-o://114e26b76f49c2a234ba1546916925d3791474ba8cc6e90d484586ee10da5b9f" gracePeriod=30 Oct 30 01:18:30.184199 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.184156 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:30.251472 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.251428 5300 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-v5jpm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:8080/ready\": dial tcp 10.133.0.76:8080: connect: connection refused" start-of-body= Oct 30 01:18:30.251658 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.251502 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" podUID="7d744ad6-75be-4a86-be0f-946122f5302e" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.76:8080/ready\": dial tcp 10.133.0.76:8080: connect: connection refused" Oct 30 01:18:30.387830 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.387799 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" Oct 30 01:18:30.443775 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.443725 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq"] Oct 30 01:18:30.455351 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:18:30.455298 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a68439c_dee6_4fb4_857d_bd09bbe8e214.slice/crio-2b259068ca24907ce5acebf5a2a744db6325d93520606300ca4e25fecb014309 WatchSource:0}: Error finding container 2b259068ca24907ce5acebf5a2a744db6325d93520606300ca4e25fecb014309: Status 404 returned error can't find the container with id 2b259068ca24907ce5acebf5a2a744db6325d93520606300ca4e25fecb014309 Oct 30 01:18:30.460642 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.460592 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:18:30.460803 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.460702 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:18:30.460803 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.460750 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:18:30.472493 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.472461 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d744ad6-75be-4a86-be0f-946122f5302e-kube-api-access-nprln\" (UniqueName: \"kubernetes.io/projected/7d744ad6-75be-4a86-be0f-946122f5302e-kube-api-access-nprln\") pod \"7d744ad6-75be-4a86-be0f-946122f5302e\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " Oct 30 01:18:30.472649 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.472516 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d744ad6-75be-4a86-be0f-946122f5302e-istio-token\" (UniqueName: \"kubernetes.io/projected/7d744ad6-75be-4a86-be0f-946122f5302e-istio-token\") pod \"7d744ad6-75be-4a86-be0f-946122f5302e\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " Oct 30 01:18:30.472649 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.472612 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d744ad6-75be-4a86-be0f-946122f5302e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7d744ad6-75be-4a86-be0f-946122f5302e-istio-kubeconfig\") pod \"7d744ad6-75be-4a86-be0f-946122f5302e\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " Oct 30 01:18:30.472755 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.472664 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d744ad6-75be-4a86-be0f-946122f5302e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7d744ad6-75be-4a86-be0f-946122f5302e-istio-csr-dns-cert\") pod \"7d744ad6-75be-4a86-be0f-946122f5302e\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " Oct 30 01:18:30.472755 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.472722 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d744ad6-75be-4a86-be0f-946122f5302e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d744ad6-75be-4a86-be0f-946122f5302e-local-certs\") pod \"7d744ad6-75be-4a86-be0f-946122f5302e\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " Oct 30 01:18:30.472940 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.472927 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d744ad6-75be-4a86-be0f-946122f5302e-cacerts\" (UniqueName: \"kubernetes.io/secret/7d744ad6-75be-4a86-be0f-946122f5302e-cacerts\") pod \"7d744ad6-75be-4a86-be0f-946122f5302e\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " Oct 30 01:18:30.473185 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.472996 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d744ad6-75be-4a86-be0f-946122f5302e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7d744ad6-75be-4a86-be0f-946122f5302e-istio-csr-ca-configmap\") pod \"7d744ad6-75be-4a86-be0f-946122f5302e\" (UID: \"7d744ad6-75be-4a86-be0f-946122f5302e\") " Oct 30 01:18:30.474130 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.474096 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d744ad6-75be-4a86-be0f-946122f5302e-istio-csr-ca-configmap" pod "7d744ad6-75be-4a86-be0f-946122f5302e" (UID: "7d744ad6-75be-4a86-be0f-946122f5302e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:30.474554 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.474519 5300 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7d744ad6-75be-4a86-be0f-946122f5302e-istio-csr-ca-configmap\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:30.482236 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.482200 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d744ad6-75be-4a86-be0f-946122f5302e-istio-csr-dns-cert" pod "7d744ad6-75be-4a86-be0f-946122f5302e" (UID: "7d744ad6-75be-4a86-be0f-946122f5302e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:18:30.482355 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.482263 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d744ad6-75be-4a86-be0f-946122f5302e-local-certs" pod "7d744ad6-75be-4a86-be0f-946122f5302e" (UID: "7d744ad6-75be-4a86-be0f-946122f5302e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:30.482355 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.482268 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d744ad6-75be-4a86-be0f-946122f5302e-cacerts" pod "7d744ad6-75be-4a86-be0f-946122f5302e" (UID: "7d744ad6-75be-4a86-be0f-946122f5302e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:18:30.482355 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.482335 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d744ad6-75be-4a86-be0f-946122f5302e-istio-kubeconfig" pod "7d744ad6-75be-4a86-be0f-946122f5302e" (UID: "7d744ad6-75be-4a86-be0f-946122f5302e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:18:30.482544 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.482480 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d744ad6-75be-4a86-be0f-946122f5302e-kube-api-access-nprln" pod "7d744ad6-75be-4a86-be0f-946122f5302e" (UID: "7d744ad6-75be-4a86-be0f-946122f5302e"). InnerVolumeSpecName "kube-api-access-nprln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:30.484011 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.483983 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d744ad6-75be-4a86-be0f-946122f5302e-istio-token" pod "7d744ad6-75be-4a86-be0f-946122f5302e" (UID: "7d744ad6-75be-4a86-be0f-946122f5302e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:30.575686 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.575647 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d744ad6-75be-4a86-be0f-946122f5302e-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:30.575686 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.575684 5300 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7d744ad6-75be-4a86-be0f-946122f5302e-istio-kubeconfig\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:30.575937 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.575699 5300 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7d744ad6-75be-4a86-be0f-946122f5302e-istio-csr-dns-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:30.575937 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.575713 5300 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d744ad6-75be-4a86-be0f-946122f5302e-local-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:30.575937 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.575727 5300 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7d744ad6-75be-4a86-be0f-946122f5302e-cacerts\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:30.575937 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.575739 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nprln\" (UniqueName: \"kubernetes.io/projected/7d744ad6-75be-4a86-be0f-946122f5302e-kube-api-access-nprln\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:30.611161 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.611123 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb"] Oct 30 01:18:30.612215 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.612193 5300 cpu_manager.go:414] "containerMap: removing container" podUID="7d744ad6-75be-4a86-be0f-946122f5302e" containerName="discovery" Oct 30 01:18:30.612267 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.612220 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d744ad6-75be-4a86-be0f-946122f5302e" containerName="discovery" Oct 30 01:18:30.612447 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.612433 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d744ad6-75be-4a86-be0f-946122f5302e" containerName="discovery" Oct 30 01:18:30.641275 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.639174 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.642838 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.642062 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb"] Oct 30 01:18:30.656180 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.656132 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-e4f8a6-openshift-default-dockercfg-sk5wh\"" Oct 30 01:18:30.657462 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.657434 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:30.677146 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.677114 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72cf531e-9c97-4a15-a1a4-0beb72c53371-istiod-ca-cert\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.677328 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.677174 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-credential-socket\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.677328 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.677206 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-podinfo\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.677328 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.677243 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-envoy\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.677512 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.677415 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-token\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.677565 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.677547 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-workload-socket\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.677617 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.677585 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5486q\" (UniqueName: \"kubernetes.io/projected/72cf531e-9c97-4a15-a1a4-0beb72c53371-kube-api-access-5486q\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.677673 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.677661 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-workload-certs\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.677931 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.677910 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-data\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.760493 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.760463 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:30.779204 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.779166 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-token\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.779389 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.779280 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-workload-socket\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.779389 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.779317 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5486q\" (UniqueName: \"kubernetes.io/projected/72cf531e-9c97-4a15-a1a4-0beb72c53371-kube-api-access-5486q\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.779389 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.779374 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-workload-certs\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.780003 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.779476 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-data\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.780003 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.779521 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72cf531e-9c97-4a15-a1a4-0beb72c53371-istiod-ca-cert\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.780003 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.779572 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-credential-socket\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.780003 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.779604 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-podinfo\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.780003 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.779639 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-envoy\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.780003 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.779755 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-workload-socket\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.780003 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.779850 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-workload-certs\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.780003 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.779994 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-data\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.780355 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.780091 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-credential-socket\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.780502 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.780480 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72cf531e-9c97-4a15-a1a4-0beb72c53371-istiod-ca-cert\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.782008 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.781984 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-envoy\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.782419 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.782395 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-podinfo\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.800392 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.800360 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-token\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.800626 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.800595 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5486q\" (UniqueName: \"kubernetes.io/projected/72cf531e-9c97-4a15-a1a4-0beb72c53371-kube-api-access-5486q\") pod \"e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:30.908690 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.908577 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:30.963311 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:30.963269 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:31.154713 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:31.154678 5300 generic.go:356] "Generic (PLEG): container finished" podID="7d744ad6-75be-4a86-be0f-946122f5302e" containerID="114e26b76f49c2a234ba1546916925d3791474ba8cc6e90d484586ee10da5b9f" exitCode=0 Oct 30 01:18:31.155246 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:31.154749 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" Oct 30 01:18:31.155246 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:31.154772 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" event={"ID":"7d744ad6-75be-4a86-be0f-946122f5302e","Type":"ContainerDied","Data":"114e26b76f49c2a234ba1546916925d3791474ba8cc6e90d484586ee10da5b9f"} Oct 30 01:18:31.155246 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:31.154859 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm" event={"ID":"7d744ad6-75be-4a86-be0f-946122f5302e","Type":"ContainerDied","Data":"493fa66a5fb35e6b746caae690849203870efb7d23f16c5dbd4232e03edb6ec4"} Oct 30 01:18:31.155246 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:31.154886 5300 scope.go:122] "RemoveContainer" containerID="114e26b76f49c2a234ba1546916925d3791474ba8cc6e90d484586ee10da5b9f" Oct 30 01:18:31.156966 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:31.156931 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" event={"ID":"2a68439c-dee6-4fb4-857d-bd09bbe8e214","Type":"ContainerStarted","Data":"0cd3d38e51e39bd6f8f0fba36c772bb4f64f3d5a09566d8b8a5fe80b821c0cd4"} Oct 30 01:18:31.157037 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:31.156990 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" event={"ID":"2a68439c-dee6-4fb4-857d-bd09bbe8e214","Type":"ContainerStarted","Data":"2b259068ca24907ce5acebf5a2a744db6325d93520606300ca4e25fecb014309"} Oct 30 01:18:31.170865 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:31.170835 5300 scope.go:122] "RemoveContainer" containerID="114e26b76f49c2a234ba1546916925d3791474ba8cc6e90d484586ee10da5b9f" Oct 30 01:18:31.171304 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:18:31.171281 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114e26b76f49c2a234ba1546916925d3791474ba8cc6e90d484586ee10da5b9f\": container with ID starting with 114e26b76f49c2a234ba1546916925d3791474ba8cc6e90d484586ee10da5b9f not found: ID does not exist" containerID="114e26b76f49c2a234ba1546916925d3791474ba8cc6e90d484586ee10da5b9f" Oct 30 01:18:31.171375 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:31.171316 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114e26b76f49c2a234ba1546916925d3791474ba8cc6e90d484586ee10da5b9f"} err="failed to get container status \"114e26b76f49c2a234ba1546916925d3791474ba8cc6e90d484586ee10da5b9f\": rpc error: code = NotFound desc = could not find container \"114e26b76f49c2a234ba1546916925d3791474ba8cc6e90d484586ee10da5b9f\": container with ID starting with 114e26b76f49c2a234ba1546916925d3791474ba8cc6e90d484586ee10da5b9f not found: ID does not exist" Oct 30 01:18:31.214406 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:31.214373 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm"] Oct 30 01:18:31.228697 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:31.228549 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v5jpm"] Oct 30 01:18:31.271802 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:31.271733 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" podStartSLOduration=2.27171801 podStartE2EDuration="2.27171801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:31.26869146 +0000 UTC m=+2111.584559805" watchObservedRunningTime="2026-10-30 01:18:31.27171801 +0000 UTC m=+2111.587586335" Oct 30 01:18:31.298868 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:31.298836 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb"] Oct 30 01:18:31.309311 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:31.309266 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:18:31.309444 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:31.309362 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:18:31.309444 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:31.309404 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:18:31.850237 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:31.850193 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d744ad6-75be-4a86-be0f-946122f5302e" path="/var/lib/kubelet/pods/7d744ad6-75be-4a86-be0f-946122f5302e/volumes" Oct 30 01:18:32.103125 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:32.103039 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:32.104573 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:32.104550 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 01:18:32.104624 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:32.104605 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" podUID="2a68439c-dee6-4fb4-857d-bd09bbe8e214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 01:18:32.165730 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:32.165695 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" event={"ID":"72cf531e-9c97-4a15-a1a4-0beb72c53371","Type":"ContainerStarted","Data":"6df25cbac4ebe899038b6e9cb904f9e50d6b8b0518d11ac5ed979b89f11b6f7d"} Oct 30 01:18:32.166187 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:32.165772 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" event={"ID":"72cf531e-9c97-4a15-a1a4-0beb72c53371","Type":"ContainerStarted","Data":"7fe0a8000980c2e8e7d1016ef693d7e36f47549b297d55816c7dc8cde1103773"} Oct 30 01:18:32.214918 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:32.214871 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" podStartSLOduration=2.214856103 podStartE2EDuration="2.214856103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:32.212171897 +0000 UTC m=+2112.528040222" watchObservedRunningTime="2026-10-30 01:18:32.214856103 +0000 UTC m=+2112.530724428" Oct 30 01:18:32.395670 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:32.395580 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:32.493733 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:32.493692 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:32.595344 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:32.595301 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:32.700103 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:32.700017 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:32.964352 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:32.964254 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:32.971840 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:32.971804 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:33.103057 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:33.103022 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 01:18:33.103262 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:33.103077 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" podUID="2a68439c-dee6-4fb4-857d-bd09bbe8e214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 01:18:33.174179 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:33.174137 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:33.174962 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:33.174942 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:18:33.691033 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:33.690991 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:34.102277 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:34.102240 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 01:18:34.102433 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:34.102293 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" podUID="2a68439c-dee6-4fb4-857d-bd09bbe8e214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 01:18:34.446055 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:34.445957 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:34.561821 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:34.561698 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:34.670705 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:34.670667 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:35.102985 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:35.102951 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 01:18:35.103154 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:35.103005 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" podUID="2a68439c-dee6-4fb4-857d-bd09bbe8e214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 01:18:36.093740 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.093648 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" Oct 30 01:18:36.102432 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.102400 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 01:18:36.102557 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.102448 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" podUID="2a68439c-dee6-4fb4-857d-bd09bbe8e214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 01:18:36.106281 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.106261 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" Oct 30 01:18:36.215580 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.215543 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" Oct 30 01:18:36.303150 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.303116 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:36.400255 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.400166 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:36.423162 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.423129 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh"] Oct 30 01:18:36.430219 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.430189 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh" Oct 30 01:18:36.449680 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.449650 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-e4f8a6\"" Oct 30 01:18:36.453541 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.453272 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh"] Oct 30 01:18:36.486380 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.486334 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm"] Oct 30 01:18:36.492008 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.491978 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm" Oct 30 01:18:36.510157 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.510133 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-e4f8a6-dockercfg-8ltn2\"" Oct 30 01:18:36.511866 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.511845 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm"] Oct 30 01:18:36.553597 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.553560 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx6kk\" (UniqueName: \"kubernetes.io/projected/31f807be-26ff-4f56-9e5b-626f349d6206-kube-api-access-xx6kk\") pod \"payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh\" (UID: \"31f807be-26ff-4f56-9e5b-626f349d6206\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh" Oct 30 01:18:36.559918 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.559892 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl8sh\" (UniqueName: \"kubernetes.io/projected/78aef360-fd10-4741-b251-30ab37a63a05-kube-api-access-tl8sh\") pod \"payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm\" (UID: \"78aef360-fd10-4741-b251-30ab37a63a05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm" Oct 30 01:18:36.563799 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.560208 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78aef360-fd10-4741-b251-30ab37a63a05-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm\" (UID: \"78aef360-fd10-4741-b251-30ab37a63a05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm" Oct 30 01:18:36.564000 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.563984 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f807be-26ff-4f56-9e5b-626f349d6206-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh\" (UID: \"31f807be-26ff-4f56-9e5b-626f349d6206\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh" Oct 30 01:18:36.666703 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.666603 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78aef360-fd10-4741-b251-30ab37a63a05-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm\" (UID: \"78aef360-fd10-4741-b251-30ab37a63a05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm" Oct 30 01:18:36.667929 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.667901 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f807be-26ff-4f56-9e5b-626f349d6206-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh\" (UID: \"31f807be-26ff-4f56-9e5b-626f349d6206\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh" Oct 30 01:18:36.668265 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.668249 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx6kk\" (UniqueName: \"kubernetes.io/projected/31f807be-26ff-4f56-9e5b-626f349d6206-kube-api-access-xx6kk\") pod \"payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh\" (UID: \"31f807be-26ff-4f56-9e5b-626f349d6206\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh" Oct 30 01:18:36.669485 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.669464 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl8sh\" (UniqueName: \"kubernetes.io/projected/78aef360-fd10-4741-b251-30ab37a63a05-kube-api-access-tl8sh\") pod \"payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm\" (UID: \"78aef360-fd10-4741-b251-30ab37a63a05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm" Oct 30 01:18:36.669966 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.669093 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f807be-26ff-4f56-9e5b-626f349d6206-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh\" (UID: \"31f807be-26ff-4f56-9e5b-626f349d6206\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh" Oct 30 01:18:36.670086 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.667806 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78aef360-fd10-4741-b251-30ab37a63a05-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm\" (UID: \"78aef360-fd10-4741-b251-30ab37a63a05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm" Oct 30 01:18:36.694789 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.694481 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx6kk\" (UniqueName: \"kubernetes.io/projected/31f807be-26ff-4f56-9e5b-626f349d6206-kube-api-access-xx6kk\") pod \"payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh\" (UID: \"31f807be-26ff-4f56-9e5b-626f349d6206\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh" Oct 30 01:18:36.695552 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.695527 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl8sh\" (UniqueName: \"kubernetes.io/projected/78aef360-fd10-4741-b251-30ab37a63a05-kube-api-access-tl8sh\") pod \"payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm\" (UID: \"78aef360-fd10-4741-b251-30ab37a63a05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm" Oct 30 01:18:36.760838 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.760814 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh" Oct 30 01:18:36.815786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.811609 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm" Oct 30 01:18:36.953890 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:36.953847 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9"] Oct 30 01:18:37.050272 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:37.050072 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:37.102541 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:37.102320 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh"] Oct 30 01:18:37.103290 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:37.102560 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 01:18:37.103290 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:37.102615 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" podUID="2a68439c-dee6-4fb4-857d-bd09bbe8e214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 01:18:37.104214 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:18:37.104165 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f807be_26ff_4f56_9e5b_626f349d6206.slice/crio-bd654e01c9022bb0effc2bb517c6820022e514fc8aa33f9458268ad5bb4baee5 WatchSource:0}: Error finding container bd654e01c9022bb0effc2bb517c6820022e514fc8aa33f9458268ad5bb4baee5: Status 404 returned error can't find the container with id bd654e01c9022bb0effc2bb517c6820022e514fc8aa33f9458268ad5bb4baee5 Oct 30 01:18:37.150204 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:37.150168 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:37.222903 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:37.222849 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh" event={"ID":"31f807be-26ff-4f56-9e5b-626f349d6206","Type":"ContainerStarted","Data":"e9766205cc9272a80f25c8a8c05fb7010231d22fb44440a4069c1c0c9c94b145"} Oct 30 01:18:37.222903 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:37.222908 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh" event={"ID":"31f807be-26ff-4f56-9e5b-626f349d6206","Type":"ContainerStarted","Data":"bd654e01c9022bb0effc2bb517c6820022e514fc8aa33f9458268ad5bb4baee5"} Oct 30 01:18:37.223148 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:37.222936 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh" Oct 30 01:18:37.249052 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:37.249023 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm"] Oct 30 01:18:37.250329 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:18:37.250299 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78aef360_fd10_4741_b251_30ab37a63a05.slice/crio-2f6c680efdf6b1b5d72b65d8856a9307cf0b0708a71a520f63de39fa2a9700ff WatchSource:0}: Error finding container 2f6c680efdf6b1b5d72b65d8856a9307cf0b0708a71a520f63de39fa2a9700ff: Status 404 returned error can't find the container with id 2f6c680efdf6b1b5d72b65d8856a9307cf0b0708a71a520f63de39fa2a9700ff Oct 30 01:18:37.264569 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:37.264526 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh" podStartSLOduration=1.264513579 podStartE2EDuration="1.264513579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:37.263273466 +0000 UTC m=+2117.579141791" watchObservedRunningTime="2026-10-30 01:18:37.264513579 +0000 UTC m=+2117.580381916" Oct 30 01:18:37.967890 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:37.967851 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p"] Oct 30 01:18:38.100082 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:38.100041 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:38.112251 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:38.112220 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:38.113794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:38.113739 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:38.114519 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:38.114498 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:18:38.198401 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:38.198365 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:38.268879 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:38.266116 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm" event={"ID":"78aef360-fd10-4741-b251-30ab37a63a05","Type":"ContainerStarted","Data":"a6b4a575a52636ded792ccad52052967af4c870dca77316a061fce24d7a4aab4"} Oct 30 01:18:38.268879 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:38.266204 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm" event={"ID":"78aef360-fd10-4741-b251-30ab37a63a05","Type":"ContainerStarted","Data":"2f6c680efdf6b1b5d72b65d8856a9307cf0b0708a71a520f63de39fa2a9700ff"} Oct 30 01:18:38.317257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:38.317198 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm" podStartSLOduration=2.317179608 podStartE2EDuration="2.317179608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:38.311572347 +0000 UTC m=+2118.627440674" watchObservedRunningTime="2026-10-30 01:18:38.317179608 +0000 UTC m=+2118.633047935" Oct 30 01:18:38.322364 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:38.322322 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:39.276823 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:39.276751 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm" Oct 30 01:18:40.003390 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:40.003356 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:41.105298 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:41.105262 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8"] Oct 30 01:18:41.105730 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:41.105478 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" podUID="3c3f3856-e654-4c1e-b1d7-35903a4dfe89" containerName="istio-proxy" containerID="cri-o://8ecc1b38721a00dd3a706be0c63b43f4cd0a34963995950d3e03dcf8cee702f7" gracePeriod=30 Oct 30 01:18:41.510027 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:41.509982 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:41.954629 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:41.954541 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9" podUID="b34e36fa-c21d-4f52-bc2e-5048d21414a7" containerName="payload-pre-processing" containerID="cri-o://7a6d0681973ab48ca0eec0f429e3acedd08eec8cace9eaa4812ca0d0bfa26905" gracePeriod=25 Oct 30 01:18:42.968198 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:42.968156 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p" podUID="0a678742-88c9-4eba-a738-3a364b1be24e" containerName="payload-processing" containerID="cri-o://6da76c9118d7ae1ebde11505a74bdf00014f0b13f6d878570cedd89a06cf4b2f" gracePeriod=25 Oct 30 01:18:43.125930 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:43.125893 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:43.854372 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:43.854325 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:43.957939 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:43.957909 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:44.743072 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:44.743034 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:45.160424 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:45.160328 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p" podUID="0a678742-88c9-4eba-a738-3a364b1be24e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:18:45.222834 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:45.222787 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:45.322244 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:45.322208 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:46.342559 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.342508 5300 generic.go:356] "Generic (PLEG): container finished" podID="3c3f3856-e654-4c1e-b1d7-35903a4dfe89" containerID="8ecc1b38721a00dd3a706be0c63b43f4cd0a34963995950d3e03dcf8cee702f7" exitCode=0 Oct 30 01:18:46.343379 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.342677 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" event={"ID":"3c3f3856-e654-4c1e-b1d7-35903a4dfe89","Type":"ContainerDied","Data":"8ecc1b38721a00dd3a706be0c63b43f4cd0a34963995950d3e03dcf8cee702f7"} Oct 30 01:18:46.385293 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.385254 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:46.415427 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.415399 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 01:18:46.452700 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.452665 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-workload-socket\") pod \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " Oct 30 01:18:46.452959 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.452729 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-credential-socket\") pod \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " Oct 30 01:18:46.452959 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.452879 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istiod-ca-cert\") pod \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " Oct 30 01:18:46.452959 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.452933 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-data\") pod \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " Oct 30 01:18:46.453139 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.452962 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-workload-socket" pod "3c3f3856-e654-4c1e-b1d7-35903a4dfe89" (UID: "3c3f3856-e654-4c1e-b1d7-35903a4dfe89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:46.453139 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.453034 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-credential-socket" pod "3c3f3856-e654-4c1e-b1d7-35903a4dfe89" (UID: "3c3f3856-e654-4c1e-b1d7-35903a4dfe89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:46.453139 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.453038 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-podinfo\") pod \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " Oct 30 01:18:46.453350 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.453158 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-data" pod "3c3f3856-e654-4c1e-b1d7-35903a4dfe89" (UID: "3c3f3856-e654-4c1e-b1d7-35903a4dfe89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:46.453350 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.453209 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-workload-certs\") pod \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " Oct 30 01:18:46.453350 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.453243 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-envoy\") pod \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " Oct 30 01:18:46.453350 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.453271 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istiod-ca-cert" pod "3c3f3856-e654-4c1e-b1d7-35903a4dfe89" (UID: "3c3f3856-e654-4c1e-b1d7-35903a4dfe89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:46.453350 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.453287 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-token\" (UniqueName: \"kubernetes.io/projected/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-token\") pod \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " Oct 30 01:18:46.453350 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.453322 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-kube-api-access-bfwfz\" (UniqueName: \"kubernetes.io/projected/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-kube-api-access-bfwfz\") pod \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\" (UID: \"3c3f3856-e654-4c1e-b1d7-35903a4dfe89\") " Oct 30 01:18:46.453844 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.453536 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-workload-certs" pod "3c3f3856-e654-4c1e-b1d7-35903a4dfe89" (UID: "3c3f3856-e654-4c1e-b1d7-35903a4dfe89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:46.454234 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.454204 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:46.454328 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.454240 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:46.454328 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.454255 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:46.454328 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.454268 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:46.454328 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.454281 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:46.455532 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.455506 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-podinfo" pod "3c3f3856-e654-4c1e-b1d7-35903a4dfe89" (UID: "3c3f3856-e654-4c1e-b1d7-35903a4dfe89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:18:46.455646 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.455564 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-kube-api-access-bfwfz" pod "3c3f3856-e654-4c1e-b1d7-35903a4dfe89" (UID: "3c3f3856-e654-4c1e-b1d7-35903a4dfe89"). InnerVolumeSpecName "kube-api-access-bfwfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:46.455646 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.455588 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-token" pod "3c3f3856-e654-4c1e-b1d7-35903a4dfe89" (UID: "3c3f3856-e654-4c1e-b1d7-35903a4dfe89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:46.455646 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.455607 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-envoy" pod "3c3f3856-e654-4c1e-b1d7-35903a4dfe89" (UID: "3c3f3856-e654-4c1e-b1d7-35903a4dfe89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:46.555440 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.555405 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:46.555440 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.555436 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:46.555440 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.555447 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:46.555712 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:46.555455 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfwfz\" (UniqueName: \"kubernetes.io/projected/3c3f3856-e654-4c1e-b1d7-35903a4dfe89-kube-api-access-bfwfz\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:18:47.178031 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:47.177992 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9" podUID="b34e36fa-c21d-4f52-bc2e-5048d21414a7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:18:47.351775 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:47.351744 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" Oct 30 01:18:47.352267 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:47.351744 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8" event={"ID":"3c3f3856-e654-4c1e-b1d7-35903a4dfe89","Type":"ContainerDied","Data":"58f74ff226bdb1015636ecc26e1f9fac9b3408a95d2d447dc809bf0a92597b60"} Oct 30 01:18:47.352267 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:47.351899 5300 scope.go:122] "RemoveContainer" containerID="8ecc1b38721a00dd3a706be0c63b43f4cd0a34963995950d3e03dcf8cee702f7" Oct 30 01:18:47.426126 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:47.426084 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8"] Oct 30 01:18:47.438842 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:47.438754 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-9x6g8"] Oct 30 01:18:47.823380 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:47.823348 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c3f3856-e654-4c1e-b1d7-35903a4dfe89" path="/var/lib/kubelet/pods/3c3f3856-e654-4c1e-b1d7-35903a4dfe89/volumes" Oct 30 01:18:47.922451 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:47.922412 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:48.270528 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:48.270497 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh" Oct 30 01:18:49.358776 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:49.358731 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:50.286813 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:50.286783 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm" Oct 30 01:18:50.809746 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:50.809712 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:51.597822 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:18:51.597752 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:18:52.445688 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:52.445656 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:53.887026 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:53.886987 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:55.160502 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:55.160459 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p" podUID="0a678742-88c9-4eba-a738-3a364b1be24e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:18:55.324326 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:55.324279 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:55.813077 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:55.813039 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:57.154544 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:57.154510 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:18:57.178724 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:57.178688 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9" podUID="b34e36fa-c21d-4f52-bc2e-5048d21414a7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:18:58.719102 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:18:58.719066 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:00.064961 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:00.064923 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:00.720187 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:00.720149 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:00.819614 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:00.819579 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:00.919687 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:00.919648 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:01.324312 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:01.324276 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:02.440954 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:02.440931 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9" Oct 30 01:19:02.480131 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:02.480099 5300 generic.go:356] "Generic (PLEG): container finished" podID="b34e36fa-c21d-4f52-bc2e-5048d21414a7" containerID="7a6d0681973ab48ca0eec0f429e3acedd08eec8cace9eaa4812ca0d0bfa26905" exitCode=0 Oct 30 01:19:02.480338 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:02.480316 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9" Oct 30 01:19:02.480534 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:02.480274 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9" event={"ID":"b34e36fa-c21d-4f52-bc2e-5048d21414a7","Type":"ContainerDied","Data":"7a6d0681973ab48ca0eec0f429e3acedd08eec8cace9eaa4812ca0d0bfa26905"} Oct 30 01:19:02.480684 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:02.480666 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9" event={"ID":"b34e36fa-c21d-4f52-bc2e-5048d21414a7","Type":"ContainerDied","Data":"93aa4c9aa0030276561fe7d7fd4cddf1097d31f49b8577e07846ea2d7fefd20a"} Oct 30 01:19:02.480817 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:02.480800 5300 scope.go:122] "RemoveContainer" containerID="7a6d0681973ab48ca0eec0f429e3acedd08eec8cace9eaa4812ca0d0bfa26905" Oct 30 01:19:02.499952 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:02.499915 5300 scope.go:122] "RemoveContainer" containerID="7a6d0681973ab48ca0eec0f429e3acedd08eec8cace9eaa4812ca0d0bfa26905" Oct 30 01:19:02.500554 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:19:02.500523 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a6d0681973ab48ca0eec0f429e3acedd08eec8cace9eaa4812ca0d0bfa26905\": container with ID starting with 7a6d0681973ab48ca0eec0f429e3acedd08eec8cace9eaa4812ca0d0bfa26905 not found: ID does not exist" containerID="7a6d0681973ab48ca0eec0f429e3acedd08eec8cace9eaa4812ca0d0bfa26905" Oct 30 01:19:02.500702 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:02.500566 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a6d0681973ab48ca0eec0f429e3acedd08eec8cace9eaa4812ca0d0bfa26905"} err="failed to get container status \"7a6d0681973ab48ca0eec0f429e3acedd08eec8cace9eaa4812ca0d0bfa26905\": rpc error: code = NotFound desc = could not find container \"7a6d0681973ab48ca0eec0f429e3acedd08eec8cace9eaa4812ca0d0bfa26905\": container with ID starting with 7a6d0681973ab48ca0eec0f429e3acedd08eec8cace9eaa4812ca0d0bfa26905 not found: ID does not exist" Oct 30 01:19:02.534509 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:02.534413 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b34e36fa-c21d-4f52-bc2e-5048d21414a7-kube-api-access-fc4s6\" (UniqueName: \"kubernetes.io/projected/b34e36fa-c21d-4f52-bc2e-5048d21414a7-kube-api-access-fc4s6\") pod \"b34e36fa-c21d-4f52-bc2e-5048d21414a7\" (UID: \"b34e36fa-c21d-4f52-bc2e-5048d21414a7\") " Oct 30 01:19:02.534509 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:02.534475 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b34e36fa-c21d-4f52-bc2e-5048d21414a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34e36fa-c21d-4f52-bc2e-5048d21414a7-plugins-config-volume\") pod \"b34e36fa-c21d-4f52-bc2e-5048d21414a7\" (UID: \"b34e36fa-c21d-4f52-bc2e-5048d21414a7\") " Oct 30 01:19:02.535735 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:02.535702 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b34e36fa-c21d-4f52-bc2e-5048d21414a7-plugins-config-volume" pod "b34e36fa-c21d-4f52-bc2e-5048d21414a7" (UID: "b34e36fa-c21d-4f52-bc2e-5048d21414a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:02.539027 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:02.538948 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b34e36fa-c21d-4f52-bc2e-5048d21414a7-kube-api-access-fc4s6" pod "b34e36fa-c21d-4f52-bc2e-5048d21414a7" (UID: "b34e36fa-c21d-4f52-bc2e-5048d21414a7"). InnerVolumeSpecName "kube-api-access-fc4s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:02.636960 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:02.636924 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc4s6\" (UniqueName: \"kubernetes.io/projected/b34e36fa-c21d-4f52-bc2e-5048d21414a7-kube-api-access-fc4s6\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:02.636960 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:02.636961 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34e36fa-c21d-4f52-bc2e-5048d21414a7-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:02.830698 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:02.830666 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9"] Oct 30 01:19:02.844003 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:02.843969 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-56d1d9-7bf84d88bc-kj8k9"] Oct 30 01:19:02.938948 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:02.938911 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:03.263887 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:03.263860 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p" Oct 30 01:19:03.347937 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:03.347850 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a678742-88c9-4eba-a738-3a364b1be24e-kube-api-access-72dj7\" (UniqueName: \"kubernetes.io/projected/0a678742-88c9-4eba-a738-3a364b1be24e-kube-api-access-72dj7\") pod \"0a678742-88c9-4eba-a738-3a364b1be24e\" (UID: \"0a678742-88c9-4eba-a738-3a364b1be24e\") " Oct 30 01:19:03.348119 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:03.347984 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a678742-88c9-4eba-a738-3a364b1be24e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a678742-88c9-4eba-a738-3a364b1be24e-plugins-config-volume\") pod \"0a678742-88c9-4eba-a738-3a364b1be24e\" (UID: \"0a678742-88c9-4eba-a738-3a364b1be24e\") " Oct 30 01:19:03.348382 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:03.348354 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a678742-88c9-4eba-a738-3a364b1be24e-plugins-config-volume" pod "0a678742-88c9-4eba-a738-3a364b1be24e" (UID: "0a678742-88c9-4eba-a738-3a364b1be24e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:03.348753 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:03.348733 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a678742-88c9-4eba-a738-3a364b1be24e-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:03.350327 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:03.350298 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a678742-88c9-4eba-a738-3a364b1be24e-kube-api-access-72dj7" pod "0a678742-88c9-4eba-a738-3a364b1be24e" (UID: "0a678742-88c9-4eba-a738-3a364b1be24e"). InnerVolumeSpecName "kube-api-access-72dj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:03.450189 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:03.450150 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72dj7\" (UniqueName: \"kubernetes.io/projected/0a678742-88c9-4eba-a738-3a364b1be24e-kube-api-access-72dj7\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:03.490365 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:03.490329 5300 generic.go:356] "Generic (PLEG): container finished" podID="0a678742-88c9-4eba-a738-3a364b1be24e" containerID="6da76c9118d7ae1ebde11505a74bdf00014f0b13f6d878570cedd89a06cf4b2f" exitCode=0 Oct 30 01:19:03.490529 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:03.490371 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p" Oct 30 01:19:03.490529 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:03.490408 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p" event={"ID":"0a678742-88c9-4eba-a738-3a364b1be24e","Type":"ContainerDied","Data":"6da76c9118d7ae1ebde11505a74bdf00014f0b13f6d878570cedd89a06cf4b2f"} Oct 30 01:19:03.490529 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:03.490460 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p" event={"ID":"0a678742-88c9-4eba-a738-3a364b1be24e","Type":"ContainerDied","Data":"380fa9279d018bb63b532a0ea1204d6fa8f689125a3f8671f7e71f1109d56c5c"} Oct 30 01:19:03.490529 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:03.490484 5300 scope.go:122] "RemoveContainer" containerID="6da76c9118d7ae1ebde11505a74bdf00014f0b13f6d878570cedd89a06cf4b2f" Oct 30 01:19:03.501733 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:03.501712 5300 scope.go:122] "RemoveContainer" containerID="6da76c9118d7ae1ebde11505a74bdf00014f0b13f6d878570cedd89a06cf4b2f" Oct 30 01:19:03.502036 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:19:03.502018 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da76c9118d7ae1ebde11505a74bdf00014f0b13f6d878570cedd89a06cf4b2f\": container with ID starting with 6da76c9118d7ae1ebde11505a74bdf00014f0b13f6d878570cedd89a06cf4b2f not found: ID does not exist" containerID="6da76c9118d7ae1ebde11505a74bdf00014f0b13f6d878570cedd89a06cf4b2f" Oct 30 01:19:03.502081 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:03.502046 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da76c9118d7ae1ebde11505a74bdf00014f0b13f6d878570cedd89a06cf4b2f"} err="failed to get container status \"6da76c9118d7ae1ebde11505a74bdf00014f0b13f6d878570cedd89a06cf4b2f\": rpc error: code = NotFound desc = could not find container \"6da76c9118d7ae1ebde11505a74bdf00014f0b13f6d878570cedd89a06cf4b2f\": container with ID starting with 6da76c9118d7ae1ebde11505a74bdf00014f0b13f6d878570cedd89a06cf4b2f not found: ID does not exist" Oct 30 01:19:03.550440 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:03.550385 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p"] Oct 30 01:19:03.554468 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:03.554436 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-82e9bf-7f866bf6dd-29g4p"] Oct 30 01:19:03.823040 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:03.823006 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a678742-88c9-4eba-a738-3a364b1be24e" path="/var/lib/kubelet/pods/0a678742-88c9-4eba-a738-3a364b1be24e/volumes" Oct 30 01:19:03.823633 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:03.823615 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b34e36fa-c21d-4f52-bc2e-5048d21414a7" path="/var/lib/kubelet/pods/b34e36fa-c21d-4f52-bc2e-5048d21414a7/volumes" Oct 30 01:19:04.301411 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:04.301367 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:05.005814 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:05.005777 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:05.106148 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:05.106115 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:05.209004 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:05.208964 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:05.636062 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:05.636028 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:06.531438 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:06.531406 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:06.643344 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:06.643304 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:06.914228 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:06.914142 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:07.372133 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:07.372096 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn"] Oct 30 01:19:07.373037 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:07.373013 5300 cpu_manager.go:414] "containerMap: removing container" podUID="b34e36fa-c21d-4f52-bc2e-5048d21414a7" containerName="payload-pre-processing" Oct 30 01:19:07.373196 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:07.373180 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b34e36fa-c21d-4f52-bc2e-5048d21414a7" containerName="payload-pre-processing" Oct 30 01:19:07.373365 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:07.373349 5300 cpu_manager.go:414] "containerMap: removing container" podUID="3c3f3856-e654-4c1e-b1d7-35903a4dfe89" containerName="istio-proxy" Oct 30 01:19:07.373478 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:07.373458 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3f3856-e654-4c1e-b1d7-35903a4dfe89" containerName="istio-proxy" Oct 30 01:19:07.373652 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:07.373487 5300 cpu_manager.go:414] "containerMap: removing container" podUID="0a678742-88c9-4eba-a738-3a364b1be24e" containerName="payload-processing" Oct 30 01:19:07.373652 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:07.373494 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a678742-88c9-4eba-a738-3a364b1be24e" containerName="payload-processing" Oct 30 01:19:07.373850 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:07.373704 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3f3856-e654-4c1e-b1d7-35903a4dfe89" containerName="istio-proxy" Oct 30 01:19:07.373850 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:07.373717 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a678742-88c9-4eba-a738-3a364b1be24e" containerName="payload-processing" Oct 30 01:19:07.373850 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:07.373744 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="b34e36fa-c21d-4f52-bc2e-5048d21414a7" containerName="payload-pre-processing" Oct 30 01:19:07.383864 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:07.383835 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn" Oct 30 01:19:07.412941 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:07.412908 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn"] Oct 30 01:19:07.435223 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:07.435188 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:07.452355 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:07.452325 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-e4f8a6-serving-cert\"" Oct 30 01:19:07.514038 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:07.513964 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c3a225-7324-4958-8533-29b9c9d2919b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn\" (UID: \"a2c3a225-7324-4958-8533-29b9c9d2919b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn" Oct 30 01:19:07.514265 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:07.514167 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c3a225-7324-4958-8533-29b9c9d2919b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn\" (UID: \"a2c3a225-7324-4958-8533-29b9c9d2919b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn" Oct 30 01:19:07.514369 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:07.514276 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c3a225-7324-4958-8533-29b9c9d2919b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn\" (UID: \"a2c3a225-7324-4958-8533-29b9c9d2919b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn" Oct 30 01:19:07.514369 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:07.514299 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp4sw\" (UniqueName: \"kubernetes.io/projected/a2c3a225-7324-4958-8533-29b9c9d2919b-kube-api-access-mp4sw\") pod \"maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn\" (UID: \"a2c3a225-7324-4958-8533-29b9c9d2919b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn" Oct 30 01:19:07.535435 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:07.535397 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:07.615193 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:07.615158 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c3a225-7324-4958-8533-29b9c9d2919b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn\" (UID: \"a2c3a225-7324-4958-8533-29b9c9d2919b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn" Oct 30 01:19:07.615375 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:07.615233 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c3a225-7324-4958-8533-29b9c9d2919b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn\" (UID: \"a2c3a225-7324-4958-8533-29b9c9d2919b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn" Oct 30 01:19:07.615375 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:07.615300 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c3a225-7324-4958-8533-29b9c9d2919b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn\" (UID: \"a2c3a225-7324-4958-8533-29b9c9d2919b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn" Oct 30 01:19:07.615375 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:07.615315 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp4sw\" (UniqueName: \"kubernetes.io/projected/a2c3a225-7324-4958-8533-29b9c9d2919b-kube-api-access-mp4sw\") pod \"maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn\" (UID: \"a2c3a225-7324-4958-8533-29b9c9d2919b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn" Oct 30 01:19:07.615562 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:19:07.615393 5300 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-serving-cert: secret "maas-api-e2e-shared-a-w3-e4f8a6-serving-cert" not found Oct 30 01:19:07.615562 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:19:07.615480 5300 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2c3a225-7324-4958-8533-29b9c9d2919b-maas-api-tls podName:a2c3a225-7324-4958-8533-29b9c9d2919b nodeName:}" failed. No retries permitted until 2026-10-30 01:19:08.115457809 +0000 UTC m=+2148.431326130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a2c3a225-7324-4958-8533-29b9c9d2919b-maas-api-tls") pod "maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn" (UID: "a2c3a225-7324-4958-8533-29b9c9d2919b") : secret "maas-api-e2e-shared-a-w3-e4f8a6-serving-cert" not found Oct 30 01:19:07.615918 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:07.615898 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c3a225-7324-4958-8533-29b9c9d2919b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn\" (UID: \"a2c3a225-7324-4958-8533-29b9c9d2919b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn" Oct 30 01:19:07.617840 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:07.617818 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c3a225-7324-4958-8533-29b9c9d2919b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn\" (UID: \"a2c3a225-7324-4958-8533-29b9c9d2919b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn" Oct 30 01:19:07.637425 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:07.637348 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp4sw\" (UniqueName: \"kubernetes.io/projected/a2c3a225-7324-4958-8533-29b9c9d2919b-kube-api-access-mp4sw\") pod \"maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn\" (UID: \"a2c3a225-7324-4958-8533-29b9c9d2919b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn" Oct 30 01:19:07.714773 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:07.714722 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:08.124779 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:08.124736 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c3a225-7324-4958-8533-29b9c9d2919b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn\" (UID: \"a2c3a225-7324-4958-8533-29b9c9d2919b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn" Oct 30 01:19:08.127222 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:08.127195 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c3a225-7324-4958-8533-29b9c9d2919b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn\" (UID: \"a2c3a225-7324-4958-8533-29b9c9d2919b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn" Oct 30 01:19:08.315169 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:08.315132 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn" Oct 30 01:19:08.324709 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:08.324665 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:08.526144 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:08.526105 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:08.638750 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:08.638710 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn"] Oct 30 01:19:08.651906 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:08.651013 5300 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:19:08.655896 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:08.655872 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr"] Oct 30 01:19:08.656116 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:08.656086 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" podUID="49cb6b12-e304-4889-b18e-2d9406bde9ac" containerName="istio-proxy" containerID="cri-o://6e19b1869bf29fb07181a0cdb247413a4290d4ef5bfe5e0dfaedca803c59ddac" gracePeriod=30 Oct 30 01:19:08.685877 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:08.685844 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:09.148533 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:09.148495 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:09.272043 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:09.272012 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:09.373717 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:09.373682 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:09.563574 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:09.563530 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn" event={"ID":"a2c3a225-7324-4958-8533-29b9c9d2919b","Type":"ContainerStarted","Data":"a236a87ea9c490ffc8a4128a37c15839a9cee6573cb77cc8f006def80c46b661"} Oct 30 01:19:09.563789 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:09.563628 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn" event={"ID":"a2c3a225-7324-4958-8533-29b9c9d2919b","Type":"ContainerStarted","Data":"89d00bd1c5106be30ebe652ac755810c16d9f2395410e3e4c32c2e4795df02b1"} Oct 30 01:19:09.563865 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:09.563803 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn" Oct 30 01:19:09.612166 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:09.612115 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn" podStartSLOduration=2.104614104 podStartE2EDuration="2.61209902s" totalImagesPullingTime="507.484916ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:09.611413336 +0000 UTC m=+2149.927281662" watchObservedRunningTime="2026-10-30 01:19:09.61209902 +0000 UTC m=+2149.927967347" Oct 30 01:19:09.839973 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:09.839884 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:10.539882 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:10.539841 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65"] Oct 30 01:19:10.540126 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:10.540096 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65" podUID="f855c133-b20f-4baa-981e-55345624598f" containerName="maas-api" containerID="cri-o://b93e8be534f921bff7973500d894d4975577a97da363591c39b40c22dde59084" gracePeriod=30 Oct 30 01:19:10.683125 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:10.683094 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:10.871015 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:10.870984 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65" Oct 30 01:19:10.998926 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:10.998885 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f855c133-b20f-4baa-981e-55345624598f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f855c133-b20f-4baa-981e-55345624598f-openshift-service-ca\") pod \"f855c133-b20f-4baa-981e-55345624598f\" (UID: \"f855c133-b20f-4baa-981e-55345624598f\") " Oct 30 01:19:10.998926 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:10.998934 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f855c133-b20f-4baa-981e-55345624598f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f855c133-b20f-4baa-981e-55345624598f-metrics-certs\") pod \"f855c133-b20f-4baa-981e-55345624598f\" (UID: \"f855c133-b20f-4baa-981e-55345624598f\") " Oct 30 01:19:10.999305 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:10.998967 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f855c133-b20f-4baa-981e-55345624598f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f855c133-b20f-4baa-981e-55345624598f-maas-api-tls\") pod \"f855c133-b20f-4baa-981e-55345624598f\" (UID: \"f855c133-b20f-4baa-981e-55345624598f\") " Oct 30 01:19:10.999305 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:10.999011 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f855c133-b20f-4baa-981e-55345624598f-kube-api-access-v79kg\" (UniqueName: \"kubernetes.io/projected/f855c133-b20f-4baa-981e-55345624598f-kube-api-access-v79kg\") pod \"f855c133-b20f-4baa-981e-55345624598f\" (UID: \"f855c133-b20f-4baa-981e-55345624598f\") " Oct 30 01:19:10.999499 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:10.999466 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f855c133-b20f-4baa-981e-55345624598f-openshift-service-ca" pod "f855c133-b20f-4baa-981e-55345624598f" (UID: "f855c133-b20f-4baa-981e-55345624598f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:10.999648 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:10.999629 5300 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f855c133-b20f-4baa-981e-55345624598f-openshift-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:11.001208 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:11.001184 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f855c133-b20f-4baa-981e-55345624598f-maas-api-tls" pod "f855c133-b20f-4baa-981e-55345624598f" (UID: "f855c133-b20f-4baa-981e-55345624598f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:11.001636 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:11.001608 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f855c133-b20f-4baa-981e-55345624598f-metrics-certs" pod "f855c133-b20f-4baa-981e-55345624598f" (UID: "f855c133-b20f-4baa-981e-55345624598f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:11.001724 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:11.001624 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f855c133-b20f-4baa-981e-55345624598f-kube-api-access-v79kg" pod "f855c133-b20f-4baa-981e-55345624598f" (UID: "f855c133-b20f-4baa-981e-55345624598f"). InnerVolumeSpecName "kube-api-access-v79kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:11.101272 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:11.101174 5300 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f855c133-b20f-4baa-981e-55345624598f-metrics-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:11.101272 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:11.101206 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f855c133-b20f-4baa-981e-55345624598f-maas-api-tls\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:11.101272 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:11.101216 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v79kg\" (UniqueName: \"kubernetes.io/projected/f855c133-b20f-4baa-981e-55345624598f-kube-api-access-v79kg\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:11.137243 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:11.137189 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:11.335032 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:11.334986 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:11.438118 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:11.438036 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:11.595807 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:11.592334 5300 generic.go:356] "Generic (PLEG): container finished" podID="f855c133-b20f-4baa-981e-55345624598f" containerID="b93e8be534f921bff7973500d894d4975577a97da363591c39b40c22dde59084" exitCode=0 Oct 30 01:19:11.595807 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:11.592411 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65" Oct 30 01:19:11.595807 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:11.592432 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65" event={"ID":"f855c133-b20f-4baa-981e-55345624598f","Type":"ContainerDied","Data":"b93e8be534f921bff7973500d894d4975577a97da363591c39b40c22dde59084"} Oct 30 01:19:11.595807 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:11.592478 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65" event={"ID":"f855c133-b20f-4baa-981e-55345624598f","Type":"ContainerDied","Data":"6a1ccfdeb6f5dec0fb9b3b1451be1948ba5f3b76ef4bd96c30cc571aeb06bc9c"} Oct 30 01:19:11.595807 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:11.592508 5300 scope.go:122] "RemoveContainer" containerID="b93e8be534f921bff7973500d894d4975577a97da363591c39b40c22dde59084" Oct 30 01:19:11.611510 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:11.611473 5300 scope.go:122] "RemoveContainer" containerID="b93e8be534f921bff7973500d894d4975577a97da363591c39b40c22dde59084" Oct 30 01:19:11.613229 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:19:11.613195 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b93e8be534f921bff7973500d894d4975577a97da363591c39b40c22dde59084\": container with ID starting with b93e8be534f921bff7973500d894d4975577a97da363591c39b40c22dde59084 not found: ID does not exist" containerID="b93e8be534f921bff7973500d894d4975577a97da363591c39b40c22dde59084" Oct 30 01:19:11.613392 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:11.613369 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b93e8be534f921bff7973500d894d4975577a97da363591c39b40c22dde59084"} err="failed to get container status \"b93e8be534f921bff7973500d894d4975577a97da363591c39b40c22dde59084\": rpc error: code = NotFound desc = could not find container \"b93e8be534f921bff7973500d894d4975577a97da363591c39b40c22dde59084\": container with ID starting with b93e8be534f921bff7973500d894d4975577a97da363591c39b40c22dde59084 not found: ID does not exist" Oct 30 01:19:11.656478 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:11.656436 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65"] Oct 30 01:19:11.668450 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:11.668417 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-34e896-565d8c884c-sjd65"] Oct 30 01:19:11.823317 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:11.823283 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f855c133-b20f-4baa-981e-55345624598f" path="/var/lib/kubelet/pods/f855c133-b20f-4baa-981e-55345624598f/volumes" Oct 30 01:19:11.912146 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:11.912111 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:12.433222 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:12.433180 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:12.537045 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:12.537002 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:12.782276 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:12.782241 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:13.298009 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:13.297976 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:13.979655 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:13.979625 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:19:14.061821 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.061781 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-workload-certs\") pod \"49cb6b12-e304-4889-b18e-2d9406bde9ac\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " Oct 30 01:19:14.062032 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.061897 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-token\" (UniqueName: \"kubernetes.io/projected/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-token\") pod \"49cb6b12-e304-4889-b18e-2d9406bde9ac\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " Oct 30 01:19:14.062032 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.062008 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-credential-socket\") pod \"49cb6b12-e304-4889-b18e-2d9406bde9ac\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " Oct 30 01:19:14.062176 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.062024 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-workload-certs" pod "49cb6b12-e304-4889-b18e-2d9406bde9ac" (UID: "49cb6b12-e304-4889-b18e-2d9406bde9ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:14.062176 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.062061 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-data\") pod \"49cb6b12-e304-4889-b18e-2d9406bde9ac\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " Oct 30 01:19:14.062176 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.062153 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49cb6b12-e304-4889-b18e-2d9406bde9ac-kube-api-access-vrvvk\" (UniqueName: \"kubernetes.io/projected/49cb6b12-e304-4889-b18e-2d9406bde9ac-kube-api-access-vrvvk\") pod \"49cb6b12-e304-4889-b18e-2d9406bde9ac\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " Oct 30 01:19:14.062340 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.062211 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-podinfo\") pod \"49cb6b12-e304-4889-b18e-2d9406bde9ac\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " Oct 30 01:19:14.062340 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.062245 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49cb6b12-e304-4889-b18e-2d9406bde9ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49cb6b12-e304-4889-b18e-2d9406bde9ac-istiod-ca-cert\") pod \"49cb6b12-e304-4889-b18e-2d9406bde9ac\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " Oct 30 01:19:14.062340 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.062293 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-credential-socket" pod "49cb6b12-e304-4889-b18e-2d9406bde9ac" (UID: "49cb6b12-e304-4889-b18e-2d9406bde9ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:14.062340 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.062314 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-workload-socket\") pod \"49cb6b12-e304-4889-b18e-2d9406bde9ac\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " Oct 30 01:19:14.062558 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.062342 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-data" pod "49cb6b12-e304-4889-b18e-2d9406bde9ac" (UID: "49cb6b12-e304-4889-b18e-2d9406bde9ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:14.062558 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.062393 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-envoy\") pod \"49cb6b12-e304-4889-b18e-2d9406bde9ac\" (UID: \"49cb6b12-e304-4889-b18e-2d9406bde9ac\") " Oct 30 01:19:14.062723 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.062674 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49cb6b12-e304-4889-b18e-2d9406bde9ac-istiod-ca-cert" pod "49cb6b12-e304-4889-b18e-2d9406bde9ac" (UID: "49cb6b12-e304-4889-b18e-2d9406bde9ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:14.063133 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.063106 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.063239 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.063138 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.063239 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.063152 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49cb6b12-e304-4889-b18e-2d9406bde9ac-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.063239 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.063164 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.063470 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.063446 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-workload-socket" pod "49cb6b12-e304-4889-b18e-2d9406bde9ac" (UID: "49cb6b12-e304-4889-b18e-2d9406bde9ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:14.065945 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.065908 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-envoy" pod "49cb6b12-e304-4889-b18e-2d9406bde9ac" (UID: "49cb6b12-e304-4889-b18e-2d9406bde9ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:14.065945 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.065933 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-token" pod "49cb6b12-e304-4889-b18e-2d9406bde9ac" (UID: "49cb6b12-e304-4889-b18e-2d9406bde9ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:14.066074 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.065940 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-podinfo" pod "49cb6b12-e304-4889-b18e-2d9406bde9ac" (UID: "49cb6b12-e304-4889-b18e-2d9406bde9ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:19:14.066074 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.065971 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49cb6b12-e304-4889-b18e-2d9406bde9ac-kube-api-access-vrvvk" pod "49cb6b12-e304-4889-b18e-2d9406bde9ac" (UID: "49cb6b12-e304-4889-b18e-2d9406bde9ac"). InnerVolumeSpecName "kube-api-access-vrvvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:14.164618 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.164581 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.164618 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.164611 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.164618 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.164622 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.164921 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.164631 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49cb6b12-e304-4889-b18e-2d9406bde9ac-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.164921 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.164643 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrvvk\" (UniqueName: \"kubernetes.io/projected/49cb6b12-e304-4889-b18e-2d9406bde9ac-kube-api-access-vrvvk\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:14.618311 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.618277 5300 generic.go:356] "Generic (PLEG): container finished" podID="49cb6b12-e304-4889-b18e-2d9406bde9ac" containerID="6e19b1869bf29fb07181a0cdb247413a4290d4ef5bfe5e0dfaedca803c59ddac" exitCode=0 Oct 30 01:19:14.618724 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.618318 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" Oct 30 01:19:14.618724 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.618351 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" event={"ID":"49cb6b12-e304-4889-b18e-2d9406bde9ac","Type":"ContainerDied","Data":"6e19b1869bf29fb07181a0cdb247413a4290d4ef5bfe5e0dfaedca803c59ddac"} Oct 30 01:19:14.618724 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.618407 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr" event={"ID":"49cb6b12-e304-4889-b18e-2d9406bde9ac","Type":"ContainerDied","Data":"a5c12d00bad3656df2e207a14deb7d1fe5cf540e1ffe35e85cd751dcb6187ff8"} Oct 30 01:19:14.618724 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.618437 5300 scope.go:122] "RemoveContainer" containerID="6e19b1869bf29fb07181a0cdb247413a4290d4ef5bfe5e0dfaedca803c59ddac" Oct 30 01:19:14.630130 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.630112 5300 scope.go:122] "RemoveContainer" containerID="65c2b209448efc4379178649cd975370cdbabd35e2e329cdea945b60591e2edd" Oct 30 01:19:14.640520 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.640498 5300 scope.go:122] "RemoveContainer" containerID="6e19b1869bf29fb07181a0cdb247413a4290d4ef5bfe5e0dfaedca803c59ddac" Oct 30 01:19:14.640788 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:19:14.640755 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e19b1869bf29fb07181a0cdb247413a4290d4ef5bfe5e0dfaedca803c59ddac\": container with ID starting with 6e19b1869bf29fb07181a0cdb247413a4290d4ef5bfe5e0dfaedca803c59ddac not found: ID does not exist" containerID="6e19b1869bf29fb07181a0cdb247413a4290d4ef5bfe5e0dfaedca803c59ddac" Oct 30 01:19:14.640882 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.640802 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e19b1869bf29fb07181a0cdb247413a4290d4ef5bfe5e0dfaedca803c59ddac"} err="failed to get container status \"6e19b1869bf29fb07181a0cdb247413a4290d4ef5bfe5e0dfaedca803c59ddac\": rpc error: code = NotFound desc = could not find container \"6e19b1869bf29fb07181a0cdb247413a4290d4ef5bfe5e0dfaedca803c59ddac\": container with ID starting with 6e19b1869bf29fb07181a0cdb247413a4290d4ef5bfe5e0dfaedca803c59ddac not found: ID does not exist" Oct 30 01:19:14.640882 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.640830 5300 scope.go:122] "RemoveContainer" containerID="65c2b209448efc4379178649cd975370cdbabd35e2e329cdea945b60591e2edd" Oct 30 01:19:14.641123 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:19:14.641081 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65c2b209448efc4379178649cd975370cdbabd35e2e329cdea945b60591e2edd\": container with ID starting with 65c2b209448efc4379178649cd975370cdbabd35e2e329cdea945b60591e2edd not found: ID does not exist" containerID="65c2b209448efc4379178649cd975370cdbabd35e2e329cdea945b60591e2edd" Oct 30 01:19:14.641167 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.641133 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65c2b209448efc4379178649cd975370cdbabd35e2e329cdea945b60591e2edd"} err="failed to get container status \"65c2b209448efc4379178649cd975370cdbabd35e2e329cdea945b60591e2edd\": rpc error: code = NotFound desc = could not find container \"65c2b209448efc4379178649cd975370cdbabd35e2e329cdea945b60591e2edd\": container with ID starting with 65c2b209448efc4379178649cd975370cdbabd35e2e329cdea945b60591e2edd not found: ID does not exist" Oct 30 01:19:14.690449 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.690420 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr"] Oct 30 01:19:14.704883 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.704852 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-82e9bf-openshift-default-666946c759-hldpr"] Oct 30 01:19:14.912993 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:14.912904 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:15.581855 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:15.581827 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn" Oct 30 01:19:15.822523 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:15.822492 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49cb6b12-e304-4889-b18e-2d9406bde9ac" path="/var/lib/kubelet/pods/49cb6b12-e304-4889-b18e-2d9406bde9ac/volumes" Oct 30 01:19:15.885683 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:15.885609 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:15.987734 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:15.987703 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:16.218090 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:16.217063 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j"] Oct 30 01:19:16.218090 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:16.217295 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" podUID="19cf7721-345d-4937-b699-292dcdde5ef0" containerName="istio-proxy" containerID="cri-o://4b07d4946a36dc0b5659d2c75e8d20ab12b204836953b8a8f44c3e1184203aa4" gracePeriod=30 Oct 30 01:19:16.245986 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:16.245945 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:16.385783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:16.384911 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:17.964977 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:17.964940 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh"] Oct 30 01:19:17.965643 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:17.965625 5300 cpu_manager.go:414] "containerMap: removing container" podUID="49cb6b12-e304-4889-b18e-2d9406bde9ac" containerName="istio-proxy" Oct 30 01:19:17.965691 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:17.965644 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49cb6b12-e304-4889-b18e-2d9406bde9ac" containerName="istio-proxy" Oct 30 01:19:17.965691 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:17.965684 5300 cpu_manager.go:414] "containerMap: removing container" podUID="f855c133-b20f-4baa-981e-55345624598f" containerName="maas-api" Oct 30 01:19:17.965691 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:17.965690 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f855c133-b20f-4baa-981e-55345624598f" containerName="maas-api" Oct 30 01:19:17.965840 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:17.965808 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="49cb6b12-e304-4889-b18e-2d9406bde9ac" containerName="istio-proxy" Oct 30 01:19:17.965840 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:17.965817 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="f855c133-b20f-4baa-981e-55345624598f" containerName="maas-api" Oct 30 01:19:17.970750 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:17.970731 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:17.989515 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:17.989482 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh"] Oct 30 01:19:17.999385 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:17.999345 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0e5711-openshift-default-dockercfg-d87hb\"" Oct 30 01:19:18.023589 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.023532 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c90a971-577a-4051-8d30-f6e46598d896-istio-token\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.023826 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.023625 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-workload-socket\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.023826 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.023690 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-istio-envoy\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.023826 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.023730 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c90a971-577a-4051-8d30-f6e46598d896-istio-podinfo\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.023826 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.023825 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-istio-data\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.024073 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.023854 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c90a971-577a-4051-8d30-f6e46598d896-istiod-ca-cert\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.024073 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.023911 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-workload-certs\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.024073 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.024030 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-credential-socket\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.024181 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.024131 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmjpq\" (UniqueName: \"kubernetes.io/projected/0c90a971-577a-4051-8d30-f6e46598d896-kube-api-access-nmjpq\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.125666 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.125608 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-istio-data\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.125666 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.125668 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c90a971-577a-4051-8d30-f6e46598d896-istiod-ca-cert\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.125997 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.125729 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-workload-certs\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.125997 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.125901 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-credential-socket\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.125997 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.125995 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmjpq\" (UniqueName: \"kubernetes.io/projected/0c90a971-577a-4051-8d30-f6e46598d896-kube-api-access-nmjpq\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.126162 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.126051 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c90a971-577a-4051-8d30-f6e46598d896-istio-token\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.126162 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.126109 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-workload-socket\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.126162 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.126134 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-istio-envoy\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.126329 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.126171 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c90a971-577a-4051-8d30-f6e46598d896-istio-podinfo\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.126329 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.126180 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-workload-certs\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.126329 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.126218 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-credential-socket\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.126485 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.126468 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-workload-socket\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.127022 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.126996 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-istio-data\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.127142 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.127128 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c90a971-577a-4051-8d30-f6e46598d896-istiod-ca-cert\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.128659 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.128634 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c90a971-577a-4051-8d30-f6e46598d896-istio-podinfo\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.128739 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.128688 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-istio-envoy\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.147504 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.147475 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmjpq\" (UniqueName: \"kubernetes.io/projected/0c90a971-577a-4051-8d30-f6e46598d896-kube-api-access-nmjpq\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.147738 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.147719 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c90a971-577a-4051-8d30-f6e46598d896-istio-token\") pod \"e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.174266 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.174229 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:18.285059 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.285018 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:18.296598 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.296569 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:18.502084 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.502045 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:18.627865 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.624837 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh"] Oct 30 01:19:18.651519 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.651471 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:19:18.651642 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.651589 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:19:18.651707 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:18.651647 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:19:19.679650 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:19.679606 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" event={"ID":"0c90a971-577a-4051-8d30-f6e46598d896","Type":"ContainerStarted","Data":"d3a0f0e20176bb3084e9712920ac98a1c75277688436c6f2d3f07752d28e87a6"} Oct 30 01:19:19.680220 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:19.679674 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" event={"ID":"0c90a971-577a-4051-8d30-f6e46598d896","Type":"ContainerStarted","Data":"55a52c4c3254be5a51ca330a8ea8dbf5aa2c4a9120f81157b261f2f3994ef056"} Oct 30 01:19:19.853948 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:19.853916 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:20.090770 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:20.090720 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:20.189718 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:20.189678 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:20.285223 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:20.285189 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:20.287086 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:20.287062 5300 patch_prober.go:29] interesting pod/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 01:19:20.287180 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:20.287110 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" podUID="0c90a971-577a-4051-8d30-f6e46598d896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 01:19:21.211418 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.211391 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:21.286331 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.286295 5300 patch_prober.go:29] interesting pod/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 01:19:21.286492 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.286366 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" podUID="0c90a971-577a-4051-8d30-f6e46598d896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 01:19:21.468901 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.468640 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:21.664988 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:19:21.664956 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:19:21.697353 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.697326 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:19:21.748940 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.748028 5300 generic.go:356] "Generic (PLEG): container finished" podID="19cf7721-345d-4937-b699-292dcdde5ef0" containerID="4b07d4946a36dc0b5659d2c75e8d20ab12b204836953b8a8f44c3e1184203aa4" exitCode=0 Oct 30 01:19:21.748940 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.748102 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" Oct 30 01:19:21.748940 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.748344 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" event={"ID":"19cf7721-345d-4937-b699-292dcdde5ef0","Type":"ContainerDied","Data":"4b07d4946a36dc0b5659d2c75e8d20ab12b204836953b8a8f44c3e1184203aa4"} Oct 30 01:19:21.748940 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.748392 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j" event={"ID":"19cf7721-345d-4937-b699-292dcdde5ef0","Type":"ContainerDied","Data":"5b106501aa15d1467af516019b9686021254f47dbbdd028d72d54c470bb08514"} Oct 30 01:19:21.748940 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.748438 5300 scope.go:122] "RemoveContainer" containerID="4b07d4946a36dc0b5659d2c75e8d20ab12b204836953b8a8f44c3e1184203aa4" Oct 30 01:19:21.758808 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.758787 5300 scope.go:122] "RemoveContainer" containerID="e80351d009fc147f9cf2d48c73e241c831642b61befc5cd9629fd51da54e82bc" Oct 30 01:19:21.768622 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.768600 5300 scope.go:122] "RemoveContainer" containerID="4b07d4946a36dc0b5659d2c75e8d20ab12b204836953b8a8f44c3e1184203aa4" Oct 30 01:19:21.768993 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:19:21.768961 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b07d4946a36dc0b5659d2c75e8d20ab12b204836953b8a8f44c3e1184203aa4\": container with ID starting with 4b07d4946a36dc0b5659d2c75e8d20ab12b204836953b8a8f44c3e1184203aa4 not found: ID does not exist" containerID="4b07d4946a36dc0b5659d2c75e8d20ab12b204836953b8a8f44c3e1184203aa4" Oct 30 01:19:21.769095 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.769003 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b07d4946a36dc0b5659d2c75e8d20ab12b204836953b8a8f44c3e1184203aa4"} err="failed to get container status \"4b07d4946a36dc0b5659d2c75e8d20ab12b204836953b8a8f44c3e1184203aa4\": rpc error: code = NotFound desc = could not find container \"4b07d4946a36dc0b5659d2c75e8d20ab12b204836953b8a8f44c3e1184203aa4\": container with ID starting with 4b07d4946a36dc0b5659d2c75e8d20ab12b204836953b8a8f44c3e1184203aa4 not found: ID does not exist" Oct 30 01:19:21.769095 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.769029 5300 scope.go:122] "RemoveContainer" containerID="e80351d009fc147f9cf2d48c73e241c831642b61befc5cd9629fd51da54e82bc" Oct 30 01:19:21.769344 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:19:21.769316 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e80351d009fc147f9cf2d48c73e241c831642b61befc5cd9629fd51da54e82bc\": container with ID starting with e80351d009fc147f9cf2d48c73e241c831642b61befc5cd9629fd51da54e82bc not found: ID does not exist" containerID="e80351d009fc147f9cf2d48c73e241c831642b61befc5cd9629fd51da54e82bc" Oct 30 01:19:21.769459 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.769356 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e80351d009fc147f9cf2d48c73e241c831642b61befc5cd9629fd51da54e82bc"} err="failed to get container status \"e80351d009fc147f9cf2d48c73e241c831642b61befc5cd9629fd51da54e82bc\": rpc error: code = NotFound desc = could not find container \"e80351d009fc147f9cf2d48c73e241c831642b61befc5cd9629fd51da54e82bc\": container with ID starting with e80351d009fc147f9cf2d48c73e241c831642b61befc5cd9629fd51da54e82bc not found: ID does not exist" Oct 30 01:19:21.784810 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.784741 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" podStartSLOduration=4.784727176 podStartE2EDuration="4.784727176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:19.74412785 +0000 UTC m=+2160.059996177" watchObservedRunningTime="2026-10-30 01:19:21.784727176 +0000 UTC m=+2162.100595500" Oct 30 01:19:21.807425 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.807394 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-istio-data\") pod \"19cf7721-345d-4937-b699-292dcdde5ef0\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " Oct 30 01:19:21.807574 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.807471 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-workload-certs\") pod \"19cf7721-345d-4937-b699-292dcdde5ef0\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " Oct 30 01:19:21.807574 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.807502 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-istio-envoy\") pod \"19cf7721-345d-4937-b699-292dcdde5ef0\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " Oct 30 01:19:21.807574 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.807545 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19cf7721-345d-4937-b699-292dcdde5ef0-istio-token\" (UniqueName: \"kubernetes.io/projected/19cf7721-345d-4937-b699-292dcdde5ef0-istio-token\") pod \"19cf7721-345d-4937-b699-292dcdde5ef0\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " Oct 30 01:19:21.807574 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.807567 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-workload-socket\") pod \"19cf7721-345d-4937-b699-292dcdde5ef0\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " Oct 30 01:19:21.807837 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.807654 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19cf7721-345d-4937-b699-292dcdde5ef0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19cf7721-345d-4937-b699-292dcdde5ef0-istiod-ca-cert\") pod \"19cf7721-345d-4937-b699-292dcdde5ef0\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " Oct 30 01:19:21.807837 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.807680 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19cf7721-345d-4937-b699-292dcdde5ef0-kube-api-access-vgk57\" (UniqueName: \"kubernetes.io/projected/19cf7721-345d-4937-b699-292dcdde5ef0-kube-api-access-vgk57\") pod \"19cf7721-345d-4937-b699-292dcdde5ef0\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " Oct 30 01:19:21.807837 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.807728 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-istio-data" pod "19cf7721-345d-4937-b699-292dcdde5ef0" (UID: "19cf7721-345d-4937-b699-292dcdde5ef0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:21.807837 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.807744 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19cf7721-345d-4937-b699-292dcdde5ef0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19cf7721-345d-4937-b699-292dcdde5ef0-istio-podinfo\") pod \"19cf7721-345d-4937-b699-292dcdde5ef0\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " Oct 30 01:19:21.807837 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.807744 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-workload-certs" pod "19cf7721-345d-4937-b699-292dcdde5ef0" (UID: "19cf7721-345d-4937-b699-292dcdde5ef0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:21.808108 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.807844 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-credential-socket\") pod \"19cf7721-345d-4937-b699-292dcdde5ef0\" (UID: \"19cf7721-345d-4937-b699-292dcdde5ef0\") " Oct 30 01:19:21.808108 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.807971 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-workload-socket" pod "19cf7721-345d-4937-b699-292dcdde5ef0" (UID: "19cf7721-345d-4937-b699-292dcdde5ef0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:21.808214 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.808194 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-credential-socket" pod "19cf7721-345d-4937-b699-292dcdde5ef0" (UID: "19cf7721-345d-4937-b699-292dcdde5ef0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:21.808638 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.808611 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:21.808754 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.808642 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:21.808754 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.808660 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:21.808754 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.808673 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:21.809039 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.808629 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19cf7721-345d-4937-b699-292dcdde5ef0-istiod-ca-cert" pod "19cf7721-345d-4937-b699-292dcdde5ef0" (UID: "19cf7721-345d-4937-b699-292dcdde5ef0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:21.810856 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.810813 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-istio-envoy" pod "19cf7721-345d-4937-b699-292dcdde5ef0" (UID: "19cf7721-345d-4937-b699-292dcdde5ef0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:21.811172 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.811152 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19cf7721-345d-4937-b699-292dcdde5ef0-istio-podinfo" pod "19cf7721-345d-4937-b699-292dcdde5ef0" (UID: "19cf7721-345d-4937-b699-292dcdde5ef0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:19:21.811449 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.811420 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19cf7721-345d-4937-b699-292dcdde5ef0-kube-api-access-vgk57" pod "19cf7721-345d-4937-b699-292dcdde5ef0" (UID: "19cf7721-345d-4937-b699-292dcdde5ef0"). InnerVolumeSpecName "kube-api-access-vgk57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:21.812536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.812515 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19cf7721-345d-4937-b699-292dcdde5ef0-istio-token" pod "19cf7721-345d-4937-b699-292dcdde5ef0" (UID: "19cf7721-345d-4937-b699-292dcdde5ef0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:21.910244 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.910205 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19cf7721-345d-4937-b699-292dcdde5ef0-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:21.910244 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.910239 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19cf7721-345d-4937-b699-292dcdde5ef0-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:21.910244 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.910249 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgk57\" (UniqueName: \"kubernetes.io/projected/19cf7721-345d-4937-b699-292dcdde5ef0-kube-api-access-vgk57\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:21.910481 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.910260 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19cf7721-345d-4937-b699-292dcdde5ef0-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:21.910481 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:21.910269 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19cf7721-345d-4937-b699-292dcdde5ef0-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:22.091647 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:22.091564 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j"] Oct 30 01:19:22.105681 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:22.105647 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-50d078-openshift-default-c45b594b7-wpd9j"] Oct 30 01:19:22.285725 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:22.285691 5300 patch_prober.go:29] interesting pod/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 01:19:22.286156 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:22.285744 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" podUID="0c90a971-577a-4051-8d30-f6e46598d896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 01:19:23.243563 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:23.243523 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:23.286025 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:23.285989 5300 patch_prober.go:29] interesting pod/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 01:19:23.286795 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:23.286046 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" podUID="0c90a971-577a-4051-8d30-f6e46598d896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 01:19:23.845540 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:23.845502 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19cf7721-345d-4937-b699-292dcdde5ef0" path="/var/lib/kubelet/pods/19cf7721-345d-4937-b699-292dcdde5ef0/volumes" Oct 30 01:19:24.286372 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:24.286339 5300 patch_prober.go:29] interesting pod/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 01:19:24.286778 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:24.286397 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" podUID="0c90a971-577a-4051-8d30-f6e46598d896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 01:19:24.843787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:24.840376 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:25.286160 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:25.286126 5300 patch_prober.go:29] interesting pod/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 01:19:25.286330 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:25.286190 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" podUID="0c90a971-577a-4051-8d30-f6e46598d896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 01:19:26.293063 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:26.293025 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:26.293573 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:26.293412 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:26.295071 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:26.295041 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:26.493259 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:26.493196 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:27.363573 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.363471 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:27.403011 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.402714 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw"] Oct 30 01:19:27.403743 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.403723 5300 cpu_manager.go:414] "containerMap: removing container" podUID="49cb6b12-e304-4889-b18e-2d9406bde9ac" containerName="istio-proxy" Oct 30 01:19:27.403831 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.403747 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49cb6b12-e304-4889-b18e-2d9406bde9ac" containerName="istio-proxy" Oct 30 01:19:27.403831 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.403800 5300 cpu_manager.go:414] "containerMap: removing container" podUID="19cf7721-345d-4937-b699-292dcdde5ef0" containerName="istio-proxy" Oct 30 01:19:27.403831 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.403809 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19cf7721-345d-4937-b699-292dcdde5ef0" containerName="istio-proxy" Oct 30 01:19:27.404107 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.403834 5300 cpu_manager.go:414] "containerMap: removing container" podUID="19cf7721-345d-4937-b699-292dcdde5ef0" containerName="istio-proxy" Oct 30 01:19:27.404107 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.403843 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19cf7721-345d-4937-b699-292dcdde5ef0" containerName="istio-proxy" Oct 30 01:19:27.404107 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.403985 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="19cf7721-345d-4937-b699-292dcdde5ef0" containerName="istio-proxy" Oct 30 01:19:27.404107 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.403997 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="19cf7721-345d-4937-b699-292dcdde5ef0" containerName="istio-proxy" Oct 30 01:19:27.404107 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.404006 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="49cb6b12-e304-4889-b18e-2d9406bde9ac" containerName="istio-proxy" Oct 30 01:19:27.409123 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.409103 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.445801 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.445294 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw"] Oct 30 01:19:27.457026 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.456530 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-69dc90-openshift-default-dockercfg-bktbh\"" Oct 30 01:19:27.463135 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.463092 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:27.561231 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.561184 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-istio-envoy\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.561444 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.561327 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-workload-certs\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.561444 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.561406 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzmgh\" (UniqueName: \"kubernetes.io/projected/488a9458-8639-4ae8-9cd4-c839b1662099-kube-api-access-fzmgh\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.561574 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.561480 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-credential-socket\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.561574 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.561547 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/488a9458-8639-4ae8-9cd4-c839b1662099-istiod-ca-cert\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.561687 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.561594 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-workload-socket\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.561687 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.561667 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/488a9458-8639-4ae8-9cd4-c839b1662099-istio-token\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.561788 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.561716 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/488a9458-8639-4ae8-9cd4-c839b1662099-istio-podinfo\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.561788 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.561739 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-istio-data\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.663448 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.663363 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-workload-certs\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.663448 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.663436 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzmgh\" (UniqueName: \"kubernetes.io/projected/488a9458-8639-4ae8-9cd4-c839b1662099-kube-api-access-fzmgh\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.663697 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.663464 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-credential-socket\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.663697 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.663487 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/488a9458-8639-4ae8-9cd4-c839b1662099-istiod-ca-cert\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.663697 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.663513 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-workload-socket\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.663697 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.663549 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/488a9458-8639-4ae8-9cd4-c839b1662099-istio-token\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.663697 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.663590 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/488a9458-8639-4ae8-9cd4-c839b1662099-istio-podinfo\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.663697 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.663620 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-istio-data\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.664094 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.663842 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-istio-envoy\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.664094 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.663858 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-workload-certs\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.664094 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.663882 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-credential-socket\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.664247 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.664197 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-workload-socket\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.664247 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.664211 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-istio-data\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.664439 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.664415 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/488a9458-8639-4ae8-9cd4-c839b1662099-istiod-ca-cert\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.666300 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.666273 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-istio-envoy\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.666539 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.666517 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/488a9458-8639-4ae8-9cd4-c839b1662099-istio-podinfo\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.684209 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.684186 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzmgh\" (UniqueName: \"kubernetes.io/projected/488a9458-8639-4ae8-9cd4-c839b1662099-kube-api-access-fzmgh\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.684481 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.684464 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/488a9458-8639-4ae8-9cd4-c839b1662099-istio-token\") pod \"e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:27.743510 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:27.743464 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:28.268642 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:19:28.268614 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod488a9458_8639_4ae8_9cd4_c839b1662099.slice/crio-5fff944e6f27a87f407b110e50e872d93f28fe763952ab32dd21ff7d6c141ef2 WatchSource:0}: Error finding container 5fff944e6f27a87f407b110e50e872d93f28fe763952ab32dd21ff7d6c141ef2: Status 404 returned error can't find the container with id 5fff944e6f27a87f407b110e50e872d93f28fe763952ab32dd21ff7d6c141ef2 Oct 30 01:19:28.271571 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:28.271535 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:19:28.271665 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:28.271606 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:19:28.271665 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:28.271641 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:19:28.274382 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:28.274357 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw"] Oct 30 01:19:28.304633 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:28.304598 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:28.804097 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:28.804060 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" event={"ID":"488a9458-8639-4ae8-9cd4-c839b1662099","Type":"ContainerStarted","Data":"0bd74398b7a2a101fb99c37b390218cd0a9b2ac659004c1cfd3075cd2516dd84"} Oct 30 01:19:28.804563 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:28.804125 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" event={"ID":"488a9458-8639-4ae8-9cd4-c839b1662099","Type":"ContainerStarted","Data":"5fff944e6f27a87f407b110e50e872d93f28fe763952ab32dd21ff7d6c141ef2"} Oct 30 01:19:29.748784 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:29.748728 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:29.754068 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:29.748911 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 01:19:29.754068 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:29.748964 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" podUID="488a9458-8639-4ae8-9cd4-c839b1662099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 01:19:29.974534 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:29.974251 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:30.744844 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:30.744216 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 01:19:30.744844 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:30.744302 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" podUID="488a9458-8639-4ae8-9cd4-c839b1662099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 01:19:31.768981 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:31.768938 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 01:19:31.769627 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:31.769587 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" podUID="488a9458-8639-4ae8-9cd4-c839b1662099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 01:19:31.937756 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:31.937720 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:32.744184 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:32.744142 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 01:19:32.744511 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:32.744474 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" podUID="488a9458-8639-4ae8-9cd4-c839b1662099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 01:19:33.639486 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:33.639445 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:33.745785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:33.744214 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 01:19:33.745785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:33.744281 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" podUID="488a9458-8639-4ae8-9cd4-c839b1662099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 01:19:33.846820 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:33.846774 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:34.215958 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:34.215911 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:19:34.216143 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:34.215981 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" podUID="72cf531e-9c97-4a15-a1a4-0beb72c53371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:19:34.744038 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:34.743998 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 01:19:34.744545 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:34.744061 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" podUID="488a9458-8639-4ae8-9cd4-c839b1662099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 01:19:35.240440 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:35.240382 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:35.749243 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:35.749213 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:35.749664 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:35.749637 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:35.750508 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:35.750488 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:19:35.826005 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:35.825955 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" podStartSLOduration=8.825939913 podStartE2EDuration="8.825939913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:28.889095011 +0000 UTC m=+2169.204963337" watchObservedRunningTime="2026-10-30 01:19:35.825939913 +0000 UTC m=+2176.141808238" Oct 30 01:19:36.554987 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:36.554956 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:36.654165 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:36.654127 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:37.112097 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:37.112057 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:37.268646 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:37.268612 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:37.369307 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:37.369270 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:37.798796 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:37.798749 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:37.955867 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:37.955835 5300 generic.go:356] "Generic (PLEG): container finished" podID="72cf531e-9c97-4a15-a1a4-0beb72c53371" containerID="6df25cbac4ebe899038b6e9cb904f9e50d6b8b0518d11ac5ed979b89f11b6f7d" exitCode=0 Oct 30 01:19:37.956049 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:37.955910 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" event={"ID":"72cf531e-9c97-4a15-a1a4-0beb72c53371","Type":"ContainerDied","Data":"6df25cbac4ebe899038b6e9cb904f9e50d6b8b0518d11ac5ed979b89f11b6f7d"} Oct 30 01:19:37.956461 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:37.956444 5300 scope.go:122] "RemoveContainer" containerID="6df25cbac4ebe899038b6e9cb904f9e50d6b8b0518d11ac5ed979b89f11b6f7d" Oct 30 01:19:37.958251 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:37.958216 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:19:37.958372 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:37.958305 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:19:37.958372 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:37.958356 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:19:38.964791 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:38.964741 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" event={"ID":"72cf531e-9c97-4a15-a1a4-0beb72c53371","Type":"ContainerStarted","Data":"9cc12a9ce996ac6873eecdaf89eaf3385e7e42a3deebbadedafc90aac161ac25"} Oct 30 01:19:39.056106 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:39.056074 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:39.744786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:39.743876 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8"] Oct 30 01:19:39.744786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:39.744143 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" podUID="6db4ab11-cb3e-4351-b252-189dc15c5578" containerName="istio-proxy" containerID="cri-o://cbcdcb80484fa0373cede25f38a1e663218f7db1e8657c00c3a746f13ad60786" gracePeriod=30 Oct 30 01:19:39.964086 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:39.964056 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:19:39.964309 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:39.964284 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 01:19:39.964398 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:39.964354 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" podUID="72cf531e-9c97-4a15-a1a4-0beb72c53371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 01:19:40.742273 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:40.742238 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:40.964375 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:40.964338 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 01:19:40.964536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:40.964391 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" podUID="72cf531e-9c97-4a15-a1a4-0beb72c53371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 01:19:41.964488 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:41.964446 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 01:19:41.964915 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:41.964509 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" podUID="72cf531e-9c97-4a15-a1a4-0beb72c53371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 01:19:42.554969 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:42.554911 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:42.964422 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:42.964331 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 01:19:42.964422 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:42.964394 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" podUID="72cf531e-9c97-4a15-a1a4-0beb72c53371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 01:19:43.846783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:43.843663 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:43.963934 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:43.963902 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 01:19:43.964126 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:43.963954 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" podUID="72cf531e-9c97-4a15-a1a4-0beb72c53371" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 01:19:44.030610 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:44.030536 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:44.149752 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:44.149344 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:44.241790 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:44.241222 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:44.350518 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:44.350437 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:44.414934 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:44.408320 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8"] Oct 30 01:19:44.414934 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:44.408556 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" podUID="c908ef75-1950-4c44-8311-47bcfd6b814a" containerName="istio-proxy" containerID="cri-o://eecc51af3bc02276f9b01844f0546ed19ca06bcb660ce94432e2bfcd2361bc3c" gracePeriod=30 Oct 30 01:19:44.461465 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:44.460049 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:44.551247 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:44.551203 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:44.954235 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:44.954176 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:45.012631 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.012316 5300 generic.go:356] "Generic (PLEG): container finished" podID="6db4ab11-cb3e-4351-b252-189dc15c5578" containerID="cbcdcb80484fa0373cede25f38a1e663218f7db1e8657c00c3a746f13ad60786" exitCode=0 Oct 30 01:19:45.012631 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.012481 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" event={"ID":"6db4ab11-cb3e-4351-b252-189dc15c5578","Type":"ContainerDied","Data":"cbcdcb80484fa0373cede25f38a1e663218f7db1e8657c00c3a746f13ad60786"} Oct 30 01:19:45.039083 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.038985 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 01:19:45.128218 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.128158 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:45.149224 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.149193 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:19:45.149620 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.149596 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:19:45.150819 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.150798 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:19:45.178403 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.178378 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-data\") pod \"6db4ab11-cb3e-4351-b252-189dc15c5578\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " Oct 30 01:19:45.178548 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.178468 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-workload-socket\") pod \"6db4ab11-cb3e-4351-b252-189dc15c5578\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " Oct 30 01:19:45.178641 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.178624 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6db4ab11-cb3e-4351-b252-189dc15c5578-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db4ab11-cb3e-4351-b252-189dc15c5578-istiod-ca-cert\") pod \"6db4ab11-cb3e-4351-b252-189dc15c5578\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " Oct 30 01:19:45.178706 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.178649 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-data" pod "6db4ab11-cb3e-4351-b252-189dc15c5578" (UID: "6db4ab11-cb3e-4351-b252-189dc15c5578"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:45.178706 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.178692 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-podinfo\") pod \"6db4ab11-cb3e-4351-b252-189dc15c5578\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " Oct 30 01:19:45.178837 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.178726 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-workload-socket" pod "6db4ab11-cb3e-4351-b252-189dc15c5578" (UID: "6db4ab11-cb3e-4351-b252-189dc15c5578"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:45.178837 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.178730 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-credential-socket\") pod \"6db4ab11-cb3e-4351-b252-189dc15c5578\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " Oct 30 01:19:45.178959 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.178844 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db4ab11-cb3e-4351-b252-189dc15c5578-kube-api-access-ktvb9\" (UniqueName: \"kubernetes.io/projected/6db4ab11-cb3e-4351-b252-189dc15c5578-kube-api-access-ktvb9\") pod \"6db4ab11-cb3e-4351-b252-189dc15c5578\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " Oct 30 01:19:45.178959 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.178915 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-workload-certs\") pod \"6db4ab11-cb3e-4351-b252-189dc15c5578\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " Oct 30 01:19:45.178959 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.178950 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-token\" (UniqueName: \"kubernetes.io/projected/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-token\") pod \"6db4ab11-cb3e-4351-b252-189dc15c5578\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " Oct 30 01:19:45.179122 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.179007 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-credential-socket" pod "6db4ab11-cb3e-4351-b252-189dc15c5578" (UID: "6db4ab11-cb3e-4351-b252-189dc15c5578"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:45.179122 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.179038 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db4ab11-cb3e-4351-b252-189dc15c5578-istiod-ca-cert" pod "6db4ab11-cb3e-4351-b252-189dc15c5578" (UID: "6db4ab11-cb3e-4351-b252-189dc15c5578"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:45.179122 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.179059 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-envoy\") pod \"6db4ab11-cb3e-4351-b252-189dc15c5578\" (UID: \"6db4ab11-cb3e-4351-b252-189dc15c5578\") " Oct 30 01:19:45.179274 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.179130 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-workload-certs" pod "6db4ab11-cb3e-4351-b252-189dc15c5578" (UID: "6db4ab11-cb3e-4351-b252-189dc15c5578"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:45.179655 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.179633 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db4ab11-cb3e-4351-b252-189dc15c5578-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:45.179755 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.179656 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:45.179755 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.179670 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:45.179755 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.179682 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:45.179755 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.179696 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:45.181226 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.181197 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-podinfo" pod "6db4ab11-cb3e-4351-b252-189dc15c5578" (UID: "6db4ab11-cb3e-4351-b252-189dc15c5578"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:19:45.181590 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.181226 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db4ab11-cb3e-4351-b252-189dc15c5578-kube-api-access-ktvb9" pod "6db4ab11-cb3e-4351-b252-189dc15c5578" (UID: "6db4ab11-cb3e-4351-b252-189dc15c5578"). InnerVolumeSpecName "kube-api-access-ktvb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:45.181590 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.181480 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-token" pod "6db4ab11-cb3e-4351-b252-189dc15c5578" (UID: "6db4ab11-cb3e-4351-b252-189dc15c5578"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:45.181590 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.181490 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-envoy" pod "6db4ab11-cb3e-4351-b252-189dc15c5578" (UID: "6db4ab11-cb3e-4351-b252-189dc15c5578"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:45.291411 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.285270 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:45.291411 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.285646 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktvb9\" (UniqueName: \"kubernetes.io/projected/6db4ab11-cb3e-4351-b252-189dc15c5578-kube-api-access-ktvb9\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:45.291411 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.285672 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:45.291411 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.285687 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db4ab11-cb3e-4351-b252-189dc15c5578-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:45.617376 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.617289 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:45.969360 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.969276 5300 patch_prober.go:29] interesting pod/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:19:45.969360 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.969333 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" podUID="6db4ab11-cb3e-4351-b252-189dc15c5578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:19:45.988395 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:45.988368 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:46.020827 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:46.020795 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" Oct 30 01:19:46.020827 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:46.020793 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8" event={"ID":"6db4ab11-cb3e-4351-b252-189dc15c5578","Type":"ContainerDied","Data":"af646ae22948040914197e6cebae0c7fa06a30fc4a4a59be3bc06c4ce6f461e9"} Oct 30 01:19:46.021089 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:46.020865 5300 scope.go:122] "RemoveContainer" containerID="cbcdcb80484fa0373cede25f38a1e663218f7db1e8657c00c3a746f13ad60786" Oct 30 01:19:46.079949 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:46.079911 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8"] Oct 30 01:19:46.093597 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:46.093563 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-zmlk8"] Oct 30 01:19:46.110590 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:46.110557 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:46.424665 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:46.424618 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:46.526242 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:46.526211 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:46.909968 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:46.909914 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:47.065785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:47.065253 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:47.822272 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:47.822238 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6db4ab11-cb3e-4351-b252-189dc15c5578" path="/var/lib/kubelet/pods/6db4ab11-cb3e-4351-b252-189dc15c5578/volumes" Oct 30 01:19:48.102813 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:48.102707 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:49.639180 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.639142 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:19:49.757154 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.757118 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-workload-socket\") pod \"c908ef75-1950-4c44-8311-47bcfd6b814a\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " Oct 30 01:19:49.757352 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.757255 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-data\") pod \"c908ef75-1950-4c44-8311-47bcfd6b814a\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " Oct 30 01:19:49.757352 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.757302 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-token\" (UniqueName: \"kubernetes.io/projected/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-token\") pod \"c908ef75-1950-4c44-8311-47bcfd6b814a\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " Oct 30 01:19:49.757495 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.757380 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-credential-socket\") pod \"c908ef75-1950-4c44-8311-47bcfd6b814a\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " Oct 30 01:19:49.757495 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.757427 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c908ef75-1950-4c44-8311-47bcfd6b814a-kube-api-access-bxf4b\" (UniqueName: \"kubernetes.io/projected/c908ef75-1950-4c44-8311-47bcfd6b814a-kube-api-access-bxf4b\") pod \"c908ef75-1950-4c44-8311-47bcfd6b814a\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " Oct 30 01:19:49.757495 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.757380 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-workload-socket" pod "c908ef75-1950-4c44-8311-47bcfd6b814a" (UID: "c908ef75-1950-4c44-8311-47bcfd6b814a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:49.757495 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.757459 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c908ef75-1950-4c44-8311-47bcfd6b814a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c908ef75-1950-4c44-8311-47bcfd6b814a-istiod-ca-cert\") pod \"c908ef75-1950-4c44-8311-47bcfd6b814a\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " Oct 30 01:19:49.757813 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.757545 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-podinfo\") pod \"c908ef75-1950-4c44-8311-47bcfd6b814a\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " Oct 30 01:19:49.757813 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.757565 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-data" pod "c908ef75-1950-4c44-8311-47bcfd6b814a" (UID: "c908ef75-1950-4c44-8311-47bcfd6b814a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:49.757813 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.757599 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-envoy\") pod \"c908ef75-1950-4c44-8311-47bcfd6b814a\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " Oct 30 01:19:49.757813 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.757682 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-credential-socket" pod "c908ef75-1950-4c44-8311-47bcfd6b814a" (UID: "c908ef75-1950-4c44-8311-47bcfd6b814a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:49.758029 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.757849 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c908ef75-1950-4c44-8311-47bcfd6b814a-istiod-ca-cert" pod "c908ef75-1950-4c44-8311-47bcfd6b814a" (UID: "c908ef75-1950-4c44-8311-47bcfd6b814a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:49.758074 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.758044 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-workload-certs\") pod \"c908ef75-1950-4c44-8311-47bcfd6b814a\" (UID: \"c908ef75-1950-4c44-8311-47bcfd6b814a\") " Oct 30 01:19:49.758074 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.758051 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-workload-certs" pod "c908ef75-1950-4c44-8311-47bcfd6b814a" (UID: "c908ef75-1950-4c44-8311-47bcfd6b814a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:49.758829 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.758806 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:49.758908 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.758837 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:49.758908 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.758852 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:49.758908 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.758867 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:49.758908 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.758880 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c908ef75-1950-4c44-8311-47bcfd6b814a-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:49.759867 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.759843 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-podinfo" pod "c908ef75-1950-4c44-8311-47bcfd6b814a" (UID: "c908ef75-1950-4c44-8311-47bcfd6b814a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:19:49.759867 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.759855 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-token" pod "c908ef75-1950-4c44-8311-47bcfd6b814a" (UID: "c908ef75-1950-4c44-8311-47bcfd6b814a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:49.760099 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.760076 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-envoy" pod "c908ef75-1950-4c44-8311-47bcfd6b814a" (UID: "c908ef75-1950-4c44-8311-47bcfd6b814a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:49.760169 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.760128 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c908ef75-1950-4c44-8311-47bcfd6b814a-kube-api-access-bxf4b" pod "c908ef75-1950-4c44-8311-47bcfd6b814a" (UID: "c908ef75-1950-4c44-8311-47bcfd6b814a"). InnerVolumeSpecName "kube-api-access-bxf4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:49.764089 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.764053 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:49.860366 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.860336 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:49.860366 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.860364 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxf4b\" (UniqueName: \"kubernetes.io/projected/c908ef75-1950-4c44-8311-47bcfd6b814a-kube-api-access-bxf4b\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:49.860366 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.860375 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:49.860642 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.860384 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c908ef75-1950-4c44-8311-47bcfd6b814a-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:49.882064 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.882018 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:49.998405 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:49.998370 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:50.091186 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:50.091102 5300 generic.go:356] "Generic (PLEG): container finished" podID="c908ef75-1950-4c44-8311-47bcfd6b814a" containerID="eecc51af3bc02276f9b01844f0546ed19ca06bcb660ce94432e2bfcd2361bc3c" exitCode=0 Oct 30 01:19:50.091186 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:50.091159 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" event={"ID":"c908ef75-1950-4c44-8311-47bcfd6b814a","Type":"ContainerDied","Data":"eecc51af3bc02276f9b01844f0546ed19ca06bcb660ce94432e2bfcd2361bc3c"} Oct 30 01:19:50.091488 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:50.091208 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" event={"ID":"c908ef75-1950-4c44-8311-47bcfd6b814a","Type":"ContainerDied","Data":"fea5702ba754a68c528e3fc9cac1da99bfa6a20556c6540d6e7375113d2b76db"} Oct 30 01:19:50.091488 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:50.091236 5300 scope.go:122] "RemoveContainer" containerID="eecc51af3bc02276f9b01844f0546ed19ca06bcb660ce94432e2bfcd2361bc3c" Oct 30 01:19:50.091488 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:50.091403 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8" Oct 30 01:19:50.102750 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:50.102729 5300 scope.go:122] "RemoveContainer" containerID="100d055f2312e10c435768d93aefdefa6a92e5c3e5682a40b472c1cda81a0e1d" Oct 30 01:19:50.113572 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:50.113552 5300 scope.go:122] "RemoveContainer" containerID="eecc51af3bc02276f9b01844f0546ed19ca06bcb660ce94432e2bfcd2361bc3c" Oct 30 01:19:50.113927 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:19:50.113900 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eecc51af3bc02276f9b01844f0546ed19ca06bcb660ce94432e2bfcd2361bc3c\": container with ID starting with eecc51af3bc02276f9b01844f0546ed19ca06bcb660ce94432e2bfcd2361bc3c not found: ID does not exist" containerID="eecc51af3bc02276f9b01844f0546ed19ca06bcb660ce94432e2bfcd2361bc3c" Oct 30 01:19:50.114004 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:50.113936 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eecc51af3bc02276f9b01844f0546ed19ca06bcb660ce94432e2bfcd2361bc3c"} err="failed to get container status \"eecc51af3bc02276f9b01844f0546ed19ca06bcb660ce94432e2bfcd2361bc3c\": rpc error: code = NotFound desc = could not find container \"eecc51af3bc02276f9b01844f0546ed19ca06bcb660ce94432e2bfcd2361bc3c\": container with ID starting with eecc51af3bc02276f9b01844f0546ed19ca06bcb660ce94432e2bfcd2361bc3c not found: ID does not exist" Oct 30 01:19:50.114004 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:50.113953 5300 scope.go:122] "RemoveContainer" containerID="100d055f2312e10c435768d93aefdefa6a92e5c3e5682a40b472c1cda81a0e1d" Oct 30 01:19:50.114242 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:19:50.114211 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"100d055f2312e10c435768d93aefdefa6a92e5c3e5682a40b472c1cda81a0e1d\": container with ID starting with 100d055f2312e10c435768d93aefdefa6a92e5c3e5682a40b472c1cda81a0e1d not found: ID does not exist" containerID="100d055f2312e10c435768d93aefdefa6a92e5c3e5682a40b472c1cda81a0e1d" Oct 30 01:19:50.114309 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:50.114253 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"100d055f2312e10c435768d93aefdefa6a92e5c3e5682a40b472c1cda81a0e1d"} err="failed to get container status \"100d055f2312e10c435768d93aefdefa6a92e5c3e5682a40b472c1cda81a0e1d\": rpc error: code = NotFound desc = could not find container \"100d055f2312e10c435768d93aefdefa6a92e5c3e5682a40b472c1cda81a0e1d\": container with ID starting with 100d055f2312e10c435768d93aefdefa6a92e5c3e5682a40b472c1cda81a0e1d not found: ID does not exist" Oct 30 01:19:50.159756 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:50.159719 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8"] Oct 30 01:19:50.167398 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:50.167367 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e69dee-openshift-default-869bf96fd9-pfvt8"] Oct 30 01:19:50.750069 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:50.749929 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:50.863408 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:50.863375 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:50.960129 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:50.960038 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:51.500356 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:51.500314 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:51.597029 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:51.596980 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:51.605065 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:19:51.604953 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:19:51.772517 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:51.772429 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:51.821841 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:51.821806 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c908ef75-1950-4c44-8311-47bcfd6b814a" path="/var/lib/kubelet/pods/c908ef75-1950-4c44-8311-47bcfd6b814a/volumes" Oct 30 01:19:51.876048 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:51.876020 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:52.246107 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:52.246063 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:52.343992 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:52.343962 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:52.558265 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:52.558173 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh"] Oct 30 01:19:52.558474 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:52.558443 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" podUID="0c90a971-577a-4051-8d30-f6e46598d896" containerName="istio-proxy" containerID="cri-o://d3a0f0e20176bb3084e9712920ac98a1c75277688436c6f2d3f07752d28e87a6" gracePeriod=30 Oct 30 01:19:52.587943 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:52.587908 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:53.348173 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:53.348134 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:53.629290 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:53.629210 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:53.875848 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:53.875817 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:54.263234 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:54.263196 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:54.365492 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:54.365459 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:55.261677 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:55.261619 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:56.296376 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:56.296339 5300 patch_prober.go:29] interesting pod/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:19:56.296826 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:56.296426 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" podUID="0c90a971-577a-4051-8d30-f6e46598d896" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:19:56.802025 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:56.801986 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:57.838563 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.838501 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:57.883993 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.883961 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-credential-socket\") pod \"0c90a971-577a-4051-8d30-f6e46598d896\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " Oct 30 01:19:57.884174 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.884043 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-istio-envoy\") pod \"0c90a971-577a-4051-8d30-f6e46598d896\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " Oct 30 01:19:57.884174 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.884100 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c90a971-577a-4051-8d30-f6e46598d896-istio-token\" (UniqueName: \"kubernetes.io/projected/0c90a971-577a-4051-8d30-f6e46598d896-istio-token\") pod \"0c90a971-577a-4051-8d30-f6e46598d896\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " Oct 30 01:19:57.884304 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.884194 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c90a971-577a-4051-8d30-f6e46598d896-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c90a971-577a-4051-8d30-f6e46598d896-istiod-ca-cert\") pod \"0c90a971-577a-4051-8d30-f6e46598d896\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " Oct 30 01:19:57.884304 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.884236 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-istio-data\") pod \"0c90a971-577a-4051-8d30-f6e46598d896\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " Oct 30 01:19:57.884304 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.884237 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-credential-socket" pod "0c90a971-577a-4051-8d30-f6e46598d896" (UID: "0c90a971-577a-4051-8d30-f6e46598d896"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:57.884304 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.884266 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-workload-certs\") pod \"0c90a971-577a-4051-8d30-f6e46598d896\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " Oct 30 01:19:57.884513 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.884302 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c90a971-577a-4051-8d30-f6e46598d896-kube-api-access-nmjpq\" (UniqueName: \"kubernetes.io/projected/0c90a971-577a-4051-8d30-f6e46598d896-kube-api-access-nmjpq\") pod \"0c90a971-577a-4051-8d30-f6e46598d896\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " Oct 30 01:19:57.884513 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.884404 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-workload-socket\") pod \"0c90a971-577a-4051-8d30-f6e46598d896\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " Oct 30 01:19:57.884513 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.884468 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-istio-data" pod "0c90a971-577a-4051-8d30-f6e46598d896" (UID: "0c90a971-577a-4051-8d30-f6e46598d896"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:57.884513 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.884493 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c90a971-577a-4051-8d30-f6e46598d896-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c90a971-577a-4051-8d30-f6e46598d896-istio-podinfo\") pod \"0c90a971-577a-4051-8d30-f6e46598d896\" (UID: \"0c90a971-577a-4051-8d30-f6e46598d896\") " Oct 30 01:19:57.884801 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.884746 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c90a971-577a-4051-8d30-f6e46598d896-istiod-ca-cert" pod "0c90a971-577a-4051-8d30-f6e46598d896" (UID: "0c90a971-577a-4051-8d30-f6e46598d896"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:57.884905 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.884889 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-workload-socket" pod "0c90a971-577a-4051-8d30-f6e46598d896" (UID: "0c90a971-577a-4051-8d30-f6e46598d896"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:57.885347 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.885316 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-workload-certs" pod "0c90a971-577a-4051-8d30-f6e46598d896" (UID: "0c90a971-577a-4051-8d30-f6e46598d896"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:57.887485 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.887453 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c90a971-577a-4051-8d30-f6e46598d896-kube-api-access-nmjpq" pod "0c90a971-577a-4051-8d30-f6e46598d896" (UID: "0c90a971-577a-4051-8d30-f6e46598d896"). InnerVolumeSpecName "kube-api-access-nmjpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:57.887594 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.887483 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-istio-envoy" pod "0c90a971-577a-4051-8d30-f6e46598d896" (UID: "0c90a971-577a-4051-8d30-f6e46598d896"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:57.887667 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.887622 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c90a971-577a-4051-8d30-f6e46598d896-istio-token" pod "0c90a971-577a-4051-8d30-f6e46598d896" (UID: "0c90a971-577a-4051-8d30-f6e46598d896"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:57.889727 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.889696 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:57.889879 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.889728 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:57.889879 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.889751 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:57.889879 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.889784 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c90a971-577a-4051-8d30-f6e46598d896-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:57.889879 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.889797 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c90a971-577a-4051-8d30-f6e46598d896-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:57.889879 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.889811 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:57.889879 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.889829 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c90a971-577a-4051-8d30-f6e46598d896-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:57.889879 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.889842 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmjpq\" (UniqueName: \"kubernetes.io/projected/0c90a971-577a-4051-8d30-f6e46598d896-kube-api-access-nmjpq\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:57.894215 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.894120 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c90a971-577a-4051-8d30-f6e46598d896-istio-podinfo" pod "0c90a971-577a-4051-8d30-f6e46598d896" (UID: "0c90a971-577a-4051-8d30-f6e46598d896"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:19:57.991246 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:57.991212 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c90a971-577a-4051-8d30-f6e46598d896-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:19:58.161175 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:58.161093 5300 generic.go:356] "Generic (PLEG): container finished" podID="0c90a971-577a-4051-8d30-f6e46598d896" containerID="d3a0f0e20176bb3084e9712920ac98a1c75277688436c6f2d3f07752d28e87a6" exitCode=0 Oct 30 01:19:58.161175 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:58.161143 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" event={"ID":"0c90a971-577a-4051-8d30-f6e46598d896","Type":"ContainerDied","Data":"d3a0f0e20176bb3084e9712920ac98a1c75277688436c6f2d3f07752d28e87a6"} Oct 30 01:19:58.161175 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:58.161164 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" Oct 30 01:19:58.161403 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:58.161207 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh" event={"ID":"0c90a971-577a-4051-8d30-f6e46598d896","Type":"ContainerDied","Data":"55a52c4c3254be5a51ca330a8ea8dbf5aa2c4a9120f81157b261f2f3994ef056"} Oct 30 01:19:58.161403 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:58.161228 5300 scope.go:122] "RemoveContainer" containerID="d3a0f0e20176bb3084e9712920ac98a1c75277688436c6f2d3f07752d28e87a6" Oct 30 01:19:58.173127 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:58.173109 5300 scope.go:122] "RemoveContainer" containerID="d3a0f0e20176bb3084e9712920ac98a1c75277688436c6f2d3f07752d28e87a6" Oct 30 01:19:58.173394 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:19:58.173370 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3a0f0e20176bb3084e9712920ac98a1c75277688436c6f2d3f07752d28e87a6\": container with ID starting with d3a0f0e20176bb3084e9712920ac98a1c75277688436c6f2d3f07752d28e87a6 not found: ID does not exist" containerID="d3a0f0e20176bb3084e9712920ac98a1c75277688436c6f2d3f07752d28e87a6" Oct 30 01:19:58.173459 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:58.173402 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3a0f0e20176bb3084e9712920ac98a1c75277688436c6f2d3f07752d28e87a6"} err="failed to get container status \"d3a0f0e20176bb3084e9712920ac98a1c75277688436c6f2d3f07752d28e87a6\": rpc error: code = NotFound desc = could not find container \"d3a0f0e20176bb3084e9712920ac98a1c75277688436c6f2d3f07752d28e87a6\": container with ID starting with d3a0f0e20176bb3084e9712920ac98a1c75277688436c6f2d3f07752d28e87a6 not found: ID does not exist" Oct 30 01:19:58.218504 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:58.218464 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh"] Oct 30 01:19:58.221465 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:58.221437 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0e5711-openshift-default-6f46b7857f-r4gxh"] Oct 30 01:19:58.351990 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:58.351956 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:58.452040 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:58.451938 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:19:59.823330 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:59.823296 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c90a971-577a-4051-8d30-f6e46598d896" path="/var/lib/kubelet/pods/0c90a971-577a-4051-8d30-f6e46598d896/volumes" Oct 30 01:19:59.998547 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:19:59.998510 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:00.099029 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:00.098936 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:00.198292 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:00.198260 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:01.618831 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:01.618785 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:03.300435 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:03.300398 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:03.404612 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:03.404577 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:04.801447 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:04.801408 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:06.454440 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:06.454401 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:08.298379 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:08.298341 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:09.696878 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:09.696835 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:11.426724 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:11.426694 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:12.931669 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:12.931631 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:14.500008 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:14.499969 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:16.020546 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:16.020508 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:17.270736 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.270701 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25"] Oct 30 01:20:17.271382 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.271365 5300 cpu_manager.go:414] "containerMap: removing container" podUID="c908ef75-1950-4c44-8311-47bcfd6b814a" containerName="istio-proxy" Oct 30 01:20:17.271438 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.271381 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c908ef75-1950-4c44-8311-47bcfd6b814a" containerName="istio-proxy" Oct 30 01:20:17.271438 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.271423 5300 cpu_manager.go:414] "containerMap: removing container" podUID="0c90a971-577a-4051-8d30-f6e46598d896" containerName="istio-proxy" Oct 30 01:20:17.271438 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.271429 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c90a971-577a-4051-8d30-f6e46598d896" containerName="istio-proxy" Oct 30 01:20:17.271539 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.271442 5300 cpu_manager.go:414] "containerMap: removing container" podUID="6db4ab11-cb3e-4351-b252-189dc15c5578" containerName="istio-proxy" Oct 30 01:20:17.271539 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.271446 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db4ab11-cb3e-4351-b252-189dc15c5578" containerName="istio-proxy" Oct 30 01:20:17.271657 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.271541 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="c908ef75-1950-4c44-8311-47bcfd6b814a" containerName="istio-proxy" Oct 30 01:20:17.271657 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.271555 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c90a971-577a-4051-8d30-f6e46598d896" containerName="istio-proxy" Oct 30 01:20:17.271657 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.271562 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="6db4ab11-cb3e-4351-b252-189dc15c5578" containerName="istio-proxy" Oct 30 01:20:17.275416 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.275394 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" Oct 30 01:20:17.306329 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.306297 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25"] Oct 30 01:20:17.310029 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.309992 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:17.311018 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.310996 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-2be695\"/\"auth-test-model-2be695-kserve-self-signed-certs\"" Oct 30 01:20:17.311352 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.311255 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-2be695\"/\"kube-root-ca.crt\"" Oct 30 01:20:17.311487 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.311392 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-2be695\"/\"openshift-service-ca.crt\"" Oct 30 01:20:17.311571 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.311441 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-2be695\"/\"default-dockercfg-dzv46\"" Oct 30 01:20:17.387530 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.387492 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-tls-certs\") pod \"auth-test-model-2be695-kserve-797888bdbf-z8m25\" (UID: \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\") " pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" Oct 30 01:20:17.387749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.387571 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-tmp-dir\") pod \"auth-test-model-2be695-kserve-797888bdbf-z8m25\" (UID: \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\") " pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" Oct 30 01:20:17.387749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.387731 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-home\") pod \"auth-test-model-2be695-kserve-797888bdbf-z8m25\" (UID: \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\") " pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" Oct 30 01:20:17.387925 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.387795 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-dshm\") pod \"auth-test-model-2be695-kserve-797888bdbf-z8m25\" (UID: \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\") " pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" Oct 30 01:20:17.387925 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.387914 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-987vg\" (UniqueName: \"kubernetes.io/projected/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-kube-api-access-987vg\") pod \"auth-test-model-2be695-kserve-797888bdbf-z8m25\" (UID: \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\") " pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" Oct 30 01:20:17.388041 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.387979 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-model-cache\") pod \"auth-test-model-2be695-kserve-797888bdbf-z8m25\" (UID: \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\") " pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" Oct 30 01:20:17.489170 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.489127 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-home\") pod \"auth-test-model-2be695-kserve-797888bdbf-z8m25\" (UID: \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\") " pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" Oct 30 01:20:17.489361 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.489182 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-dshm\") pod \"auth-test-model-2be695-kserve-797888bdbf-z8m25\" (UID: \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\") " pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" Oct 30 01:20:17.489361 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.489254 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-987vg\" (UniqueName: \"kubernetes.io/projected/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-kube-api-access-987vg\") pod \"auth-test-model-2be695-kserve-797888bdbf-z8m25\" (UID: \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\") " pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" Oct 30 01:20:17.489361 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.489303 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-model-cache\") pod \"auth-test-model-2be695-kserve-797888bdbf-z8m25\" (UID: \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\") " pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" Oct 30 01:20:17.489544 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.489378 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-tls-certs\") pod \"auth-test-model-2be695-kserve-797888bdbf-z8m25\" (UID: \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\") " pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" Oct 30 01:20:17.489544 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.489411 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-tmp-dir\") pod \"auth-test-model-2be695-kserve-797888bdbf-z8m25\" (UID: \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\") " pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" Oct 30 01:20:17.489679 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.489636 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-home\") pod \"auth-test-model-2be695-kserve-797888bdbf-z8m25\" (UID: \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\") " pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" Oct 30 01:20:17.489907 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.489831 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-tmp-dir\") pod \"auth-test-model-2be695-kserve-797888bdbf-z8m25\" (UID: \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\") " pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" Oct 30 01:20:17.490028 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.489960 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-model-cache\") pod \"auth-test-model-2be695-kserve-797888bdbf-z8m25\" (UID: \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\") " pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" Oct 30 01:20:17.491860 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.491835 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-dshm\") pod \"auth-test-model-2be695-kserve-797888bdbf-z8m25\" (UID: \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\") " pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" Oct 30 01:20:17.492369 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.492346 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-tls-certs\") pod \"auth-test-model-2be695-kserve-797888bdbf-z8m25\" (UID: \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\") " pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" Oct 30 01:20:17.530734 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.530624 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-987vg\" (UniqueName: \"kubernetes.io/projected/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-kube-api-access-987vg\") pod \"auth-test-model-2be695-kserve-797888bdbf-z8m25\" (UID: \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\") " pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" Oct 30 01:20:17.531774 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.531723 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj"] Oct 30 01:20:17.533040 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.533013 5300 cpu_manager.go:414] "containerMap: removing container" podUID="c908ef75-1950-4c44-8311-47bcfd6b814a" containerName="istio-proxy" Oct 30 01:20:17.533040 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.533040 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c908ef75-1950-4c44-8311-47bcfd6b814a" containerName="istio-proxy" Oct 30 01:20:17.533297 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.533279 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="c908ef75-1950-4c44-8311-47bcfd6b814a" containerName="istio-proxy" Oct 30 01:20:17.540449 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.540427 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj" Oct 30 01:20:17.559070 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.557487 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-e2d6d3-serving-cert\"" Oct 30 01:20:17.562038 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.562004 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj"] Oct 30 01:20:17.590701 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.590661 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd43904-bf14-4a9c-a752-d18c1fcff864-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj\" (UID: \"ffd43904-bf14-4a9c-a752-d18c1fcff864\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj" Oct 30 01:20:17.590968 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.590947 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffd43904-bf14-4a9c-a752-d18c1fcff864-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj\" (UID: \"ffd43904-bf14-4a9c-a752-d18c1fcff864\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj" Oct 30 01:20:17.591250 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.591231 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxvwd\" (UniqueName: \"kubernetes.io/projected/ffd43904-bf14-4a9c-a752-d18c1fcff864-kube-api-access-bxvwd\") pod \"maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj\" (UID: \"ffd43904-bf14-4a9c-a752-d18c1fcff864\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj" Oct 30 01:20:17.591465 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.591446 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffd43904-bf14-4a9c-a752-d18c1fcff864-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj\" (UID: \"ffd43904-bf14-4a9c-a752-d18c1fcff864\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj" Oct 30 01:20:17.591696 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.590885 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" Oct 30 01:20:17.614319 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.614285 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:17.694166 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.692678 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxvwd\" (UniqueName: \"kubernetes.io/projected/ffd43904-bf14-4a9c-a752-d18c1fcff864-kube-api-access-bxvwd\") pod \"maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj\" (UID: \"ffd43904-bf14-4a9c-a752-d18c1fcff864\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj" Oct 30 01:20:17.694166 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.692830 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffd43904-bf14-4a9c-a752-d18c1fcff864-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj\" (UID: \"ffd43904-bf14-4a9c-a752-d18c1fcff864\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj" Oct 30 01:20:17.694166 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.692976 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd43904-bf14-4a9c-a752-d18c1fcff864-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj\" (UID: \"ffd43904-bf14-4a9c-a752-d18c1fcff864\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj" Oct 30 01:20:17.694166 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.693000 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffd43904-bf14-4a9c-a752-d18c1fcff864-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj\" (UID: \"ffd43904-bf14-4a9c-a752-d18c1fcff864\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj" Oct 30 01:20:17.694166 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:20:17.693224 5300 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-serving-cert: secret "maas-api-e2e-shared-a-w4-e2d6d3-serving-cert" not found Oct 30 01:20:17.694166 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:20:17.693289 5300 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffd43904-bf14-4a9c-a752-d18c1fcff864-maas-api-tls podName:ffd43904-bf14-4a9c-a752-d18c1fcff864 nodeName:}" failed. No retries permitted until 2026-10-30 01:20:18.193268128 +0000 UTC m=+2218.509136433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ffd43904-bf14-4a9c-a752-d18c1fcff864-maas-api-tls") pod "maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj" (UID: "ffd43904-bf14-4a9c-a752-d18c1fcff864") : secret "maas-api-e2e-shared-a-w4-e2d6d3-serving-cert" not found Oct 30 01:20:17.694684 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.694649 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd43904-bf14-4a9c-a752-d18c1fcff864-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj\" (UID: \"ffd43904-bf14-4a9c-a752-d18c1fcff864\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj" Oct 30 01:20:17.696856 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.696830 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffd43904-bf14-4a9c-a752-d18c1fcff864-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj\" (UID: \"ffd43904-bf14-4a9c-a752-d18c1fcff864\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj" Oct 30 01:20:17.713226 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.713187 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:17.717517 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.717491 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxvwd\" (UniqueName: \"kubernetes.io/projected/ffd43904-bf14-4a9c-a752-d18c1fcff864-kube-api-access-bxvwd\") pod \"maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj\" (UID: \"ffd43904-bf14-4a9c-a752-d18c1fcff864\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj" Oct 30 01:20:17.905102 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:17.905067 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25"] Oct 30 01:20:18.041551 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:18.041486 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:18.200577 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:18.200480 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffd43904-bf14-4a9c-a752-d18c1fcff864-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj\" (UID: \"ffd43904-bf14-4a9c-a752-d18c1fcff864\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj" Oct 30 01:20:18.202713 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:18.202693 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffd43904-bf14-4a9c-a752-d18c1fcff864-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj\" (UID: \"ffd43904-bf14-4a9c-a752-d18c1fcff864\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj" Oct 30 01:20:18.321408 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:18.321371 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" event={"ID":"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f","Type":"ContainerStarted","Data":"8e79610fea6373f5170e50c87e7b9f7ec69a0757f6ab635902da1d2d6fd00b82"} Oct 30 01:20:18.321823 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:18.321417 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" event={"ID":"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f","Type":"ContainerStarted","Data":"a3ca1e847fffdc014176c75048f2ec93b97a1cccaf234554ab3af5a311c8f745"} Oct 30 01:20:18.364665 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:18.364608 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" podStartSLOduration=1.364592644 podStartE2EDuration="1.364592644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:18.362528144 +0000 UTC m=+2218.678396471" watchObservedRunningTime="2026-10-30 01:20:18.364592644 +0000 UTC m=+2218.680460999" Oct 30 01:20:18.466171 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:18.466076 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj" Oct 30 01:20:18.592361 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:18.592328 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" Oct 30 01:20:18.804354 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:18.804325 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj"] Oct 30 01:20:18.805927 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:20:18.805887 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd43904_bf14_4a9c_a752_d18c1fcff864.slice/crio-d15f178ec6ed6de899349737f31c5a6f0cc7181462dc46e0e970ce19a0e09298 WatchSource:0}: Error finding container d15f178ec6ed6de899349737f31c5a6f0cc7181462dc46e0e970ce19a0e09298: Status 404 returned error can't find the container with id d15f178ec6ed6de899349737f31c5a6f0cc7181462dc46e0e970ce19a0e09298 Oct 30 01:20:19.318818 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:19.318739 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:19.330177 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:19.330142 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj" event={"ID":"ffd43904-bf14-4a9c-a752-d18c1fcff864","Type":"ContainerStarted","Data":"80abc9bf42fd6b7b9d2af876a523c782b588b9a94e5af7de55cadb5d68a92713"} Oct 30 01:20:19.330613 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:19.330231 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj" event={"ID":"ffd43904-bf14-4a9c-a752-d18c1fcff864","Type":"ContainerStarted","Data":"d15f178ec6ed6de899349737f31c5a6f0cc7181462dc46e0e970ce19a0e09298"} Oct 30 01:20:19.330613 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:19.330442 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj" Oct 30 01:20:19.374179 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:19.374072 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj" podStartSLOduration=2.083485189 podStartE2EDuration="2.37405783s" totalImagesPullingTime="290.572641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:19.373160765 +0000 UTC m=+2219.689029084" watchObservedRunningTime="2026-10-30 01:20:19.37405783 +0000 UTC m=+2219.689926154" Oct 30 01:20:19.419996 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:19.419965 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:19.522810 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:19.522772 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:20.834090 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:20.834051 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:21.594185 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:20:21.594143 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:20:22.250466 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:22.250433 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:23.096123 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.096087 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5"] Oct 30 01:20:23.103003 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.102984 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.121496 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.121472 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5"] Oct 30 01:20:23.131583 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.131556 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-18a189-openshift-default-dockercfg-rcm79\"" Oct 30 01:20:23.175617 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.175584 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-credential-socket\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.175809 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.175626 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsv6t\" (UniqueName: \"kubernetes.io/projected/cfe18c62-795e-43b0-826c-35b3e91d9890-kube-api-access-xsv6t\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.175809 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.175701 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-envoy\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.175809 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.175739 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-podinfo\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.175939 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.175868 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-token\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.175939 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.175903 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-workload-socket\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.175939 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.175934 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-data\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.176036 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.175994 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-workload-certs\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.176085 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.176036 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe18c62-795e-43b0-826c-35b3e91d9890-istiod-ca-cert\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.277549 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.277510 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-credential-socket\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.278077 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.277562 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsv6t\" (UniqueName: \"kubernetes.io/projected/cfe18c62-795e-43b0-826c-35b3e91d9890-kube-api-access-xsv6t\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.278077 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.277604 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-envoy\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.278077 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.277659 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-podinfo\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.278077 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.277756 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-token\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.278077 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.277842 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-workload-socket\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.278077 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.277893 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-data\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.278077 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.277967 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-credential-socket\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.278077 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.277976 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-workload-certs\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.278077 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.278070 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe18c62-795e-43b0-826c-35b3e91d9890-istiod-ca-cert\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.278511 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.278224 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-workload-socket\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.278511 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.278261 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-data\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.278511 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.278319 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-workload-certs\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.278867 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.278843 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe18c62-795e-43b0-826c-35b3e91d9890-istiod-ca-cert\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.280226 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.280199 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-envoy\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.280437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.280412 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-podinfo\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.298245 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.298215 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-token\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.298414 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.298391 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsv6t\" (UniqueName: \"kubernetes.io/projected/cfe18c62-795e-43b0-826c-35b3e91d9890-kube-api-access-xsv6t\") pod \"e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.417941 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.417857 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:23.737925 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.737892 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5"] Oct 30 01:20:23.742325 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:20:23.742299 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe18c62_795e_43b0_826c_35b3e91d9890.slice/crio-7c1ba0d8fac3b748705ca0ac348c1af584a3a3b359b119e429fba87f66e92d04 WatchSource:0}: Error finding container 7c1ba0d8fac3b748705ca0ac348c1af584a3a3b359b119e429fba87f66e92d04: Status 404 returned error can't find the container with id 7c1ba0d8fac3b748705ca0ac348c1af584a3a3b359b119e429fba87f66e92d04 Oct 30 01:20:23.746120 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.746084 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:20:23.746200 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.746154 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:20:23.746200 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.746183 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:20:23.822677 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:23.822650 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:24.376943 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:24.376902 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" event={"ID":"cfe18c62-795e-43b0-826c-35b3e91d9890","Type":"ContainerStarted","Data":"93c02683d1aa5fc4039b3aa66b43c7e274645fa4ca61129446b39492efdba20c"} Oct 30 01:20:24.377365 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:24.376965 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" event={"ID":"cfe18c62-795e-43b0-826c-35b3e91d9890","Type":"ContainerStarted","Data":"7c1ba0d8fac3b748705ca0ac348c1af584a3a3b359b119e429fba87f66e92d04"} Oct 30 01:20:24.424785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:24.422080 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:24.442788 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:24.429522 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 01:20:24.442788 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:24.429586 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" podUID="cfe18c62-795e-43b0-826c-35b3e91d9890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 01:20:24.446444 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:24.446388 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" podStartSLOduration=1.4463725649999999 podStartE2EDuration="1.446372565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:24.444523629 +0000 UTC m=+2224.760391956" watchObservedRunningTime="2026-10-30 01:20:24.446372565 +0000 UTC m=+2224.762240889" Oct 30 01:20:25.343302 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:25.343268 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:25.348965 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:25.348943 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj" Oct 30 01:20:25.443816 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:25.426108 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 01:20:25.443816 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:25.426164 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" podUID="cfe18c62-795e-43b0-826c-35b3e91d9890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 01:20:25.447171 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:25.447123 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:26.420207 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:26.419701 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 01:20:26.420207 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:26.419756 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" podUID="cfe18c62-795e-43b0-826c-35b3e91d9890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 01:20:27.125277 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:27.125235 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:27.423843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:27.420160 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 01:20:27.423843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:27.420239 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" podUID="cfe18c62-795e-43b0-826c-35b3e91d9890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 01:20:27.592495 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:27.592458 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" Oct 30 01:20:27.606585 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:27.606553 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" Oct 30 01:20:28.417449 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:28.417422 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" Oct 30 01:20:28.423120 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:28.423089 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 01:20:28.436782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:28.424384 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" podUID="cfe18c62-795e-43b0-826c-35b3e91d9890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 01:20:28.909104 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:28.909066 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:29.010169 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:29.010125 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:29.111313 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:29.111273 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:29.437783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:29.436116 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 30 01:20:29.437783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:29.436189 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" podUID="cfe18c62-795e-43b0-826c-35b3e91d9890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 30 01:20:30.423072 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:30.423044 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:30.423513 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:30.423494 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:30.424219 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:30.424200 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:20:30.442111 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:30.442081 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:31.978369 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:31.978336 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:33.524588 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:33.524545 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:33.968171 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:33.968076 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:34.469107 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:34.469068 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw"] Oct 30 01:20:34.494339 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:34.494303 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:35.119899 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:35.119867 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:36.522093 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:36.522057 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:36.888688 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:36.888598 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz"] Oct 30 01:20:36.894257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:36.894228 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:36.940306 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:36.940262 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz"] Oct 30 01:20:36.940508 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:36.940453 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-29145c-openshift-default-dockercfg-5nhvg\"" Oct 30 01:20:36.968393 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:36.968363 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:37.002243 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.002196 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8749ea6-d001-45fd-b7bd-956be5364f8c-istiod-ca-cert\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.002438 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.002252 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-credential-socket\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.002438 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.002368 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-data\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.002438 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.002426 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-envoy\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.002562 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.002538 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-workload-certs\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.002617 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.002603 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-workload-socket\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.002656 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.002626 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpjfl\" (UniqueName: \"kubernetes.io/projected/c8749ea6-d001-45fd-b7bd-956be5364f8c-kube-api-access-wpjfl\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.002656 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.002650 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-podinfo\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.002751 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.002676 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-token\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.104109 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.104072 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-data\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.104328 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.104161 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-envoy\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.104328 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.104276 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-workload-certs\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.104465 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.104383 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-workload-socket\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.104465 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.104408 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpjfl\" (UniqueName: \"kubernetes.io/projected/c8749ea6-d001-45fd-b7bd-956be5364f8c-kube-api-access-wpjfl\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.104465 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.104441 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-podinfo\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.104633 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.104480 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-token\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.104633 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.104491 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-data\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.104912 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.104889 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-workload-socket\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.105112 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.105020 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-workload-certs\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.105224 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.105030 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8749ea6-d001-45fd-b7bd-956be5364f8c-istiod-ca-cert\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.105359 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.105255 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-credential-socket\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.105610 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.105555 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8749ea6-d001-45fd-b7bd-956be5364f8c-istiod-ca-cert\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.105740 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.105714 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-credential-socket\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.107287 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.107264 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-envoy\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.107641 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.107618 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-podinfo\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.128817 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.128745 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-token\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.128963 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.128746 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpjfl\" (UniqueName: \"kubernetes.io/projected/c8749ea6-d001-45fd-b7bd-956be5364f8c-kube-api-access-wpjfl\") pod \"e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.211433 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.211349 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:37.531961 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.531925 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz"] Oct 30 01:20:37.533321 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:20:37.533293 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8749ea6_d001_45fd_b7bd_956be5364f8c.slice/crio-d0268dfa43c4eb3942f5f6b7724165408cc60c2df6fc00c0dcb07d2cf2310072 WatchSource:0}: Error finding container d0268dfa43c4eb3942f5f6b7724165408cc60c2df6fc00c0dcb07d2cf2310072: Status 404 returned error can't find the container with id d0268dfa43c4eb3942f5f6b7724165408cc60c2df6fc00c0dcb07d2cf2310072 Oct 30 01:20:37.536242 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.536205 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:20:37.536328 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.536276 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:20:37.536328 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.536315 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:20:37.554277 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:37.554241 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:38.006099 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:38.006061 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:38.181375 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:38.181336 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:38.281150 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:38.281068 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:38.473189 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:38.473156 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" event={"ID":"c8749ea6-d001-45fd-b7bd-956be5364f8c","Type":"ContainerStarted","Data":"8c21f52f41747dc8404c42511b14dc856a04a11f9da3129fcd6522253e2f8e1b"} Oct 30 01:20:38.473463 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:38.473432 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" event={"ID":"c8749ea6-d001-45fd-b7bd-956be5364f8c","Type":"ContainerStarted","Data":"d0268dfa43c4eb3942f5f6b7724165408cc60c2df6fc00c0dcb07d2cf2310072"} Oct 30 01:20:38.563398 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:38.563292 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" podStartSLOduration=2.563275674 podStartE2EDuration="2.563275674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:38.548255543 +0000 UTC m=+2238.864123869" watchObservedRunningTime="2026-10-30 01:20:38.563275674 +0000 UTC m=+2238.879143999" Oct 30 01:20:39.212388 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:39.212357 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:39.213907 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:39.213871 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:20:39.214050 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:39.213919 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" podUID="c8749ea6-d001-45fd-b7bd-956be5364f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 01:20:39.780801 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:39.780733 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:40.211985 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:40.211898 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:20:40.211985 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:40.211959 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" podUID="c8749ea6-d001-45fd-b7bd-956be5364f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 01:20:41.212247 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:41.212214 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:20:41.212620 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:41.212267 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" podUID="c8749ea6-d001-45fd-b7bd-956be5364f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 01:20:41.442841 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:41.442425 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:41.801031 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:41.800999 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:41.896613 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:41.896579 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:42.211909 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:42.211820 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:20:42.211909 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:42.211892 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" podUID="c8749ea6-d001-45fd-b7bd-956be5364f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 01:20:43.212354 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:43.212322 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:20:43.212750 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:43.212387 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" podUID="c8749ea6-d001-45fd-b7bd-956be5364f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 01:20:43.287060 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:43.287023 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:43.503735 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:43.503694 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:43.599983 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:43.599949 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:44.211952 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:44.211902 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:20:44.212147 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:44.211969 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" podUID="c8749ea6-d001-45fd-b7bd-956be5364f8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 01:20:45.131461 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:45.131421 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:45.217547 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:45.217517 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:45.217845 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:45.217816 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:45.218618 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:45.218595 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:20:46.876592 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:46.876554 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:48.435455 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:48.435405 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:49.490747 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:49.490704 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" podUID="ee55bf23-0c5c-4078-9149-ff5a6fab2c3d" containerName="main" containerID="cri-o://7fb2d755c9a5f4f70d52242ce2527ccfc17f70a648127975e4e1b8106746bc59" gracePeriod=45 Oct 30 01:20:49.777470 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:49.777439 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" Oct 30 01:20:49.901301 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:49.901268 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-dshm\") pod \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\" (UID: \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\") " Oct 30 01:20:49.901466 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:49.901424 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-model-cache\") pod \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\" (UID: \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\") " Oct 30 01:20:49.901544 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:49.901534 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-tmp-dir\") pod \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\" (UID: \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\") " Oct 30 01:20:49.901611 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:49.901595 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-tls-certs\") pod \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\" (UID: \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\") " Oct 30 01:20:49.901660 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:49.901633 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-kube-api-access-7vsnw\" (UniqueName: \"kubernetes.io/projected/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-kube-api-access-7vsnw\") pod \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\" (UID: \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\") " Oct 30 01:20:49.901709 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:49.901678 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-home\" (UniqueName: \"kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-home\") pod \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\" (UID: \"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d\") " Oct 30 01:20:49.901802 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:49.901729 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-model-cache" pod "ee55bf23-0c5c-4078-9149-ff5a6fab2c3d" (UID: "ee55bf23-0c5c-4078-9149-ff5a6fab2c3d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:49.901923 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:49.901800 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-tmp-dir" pod "ee55bf23-0c5c-4078-9149-ff5a6fab2c3d" (UID: "ee55bf23-0c5c-4078-9149-ff5a6fab2c3d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:49.902045 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:49.902014 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-home" pod "ee55bf23-0c5c-4078-9149-ff5a6fab2c3d" (UID: "ee55bf23-0c5c-4078-9149-ff5a6fab2c3d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:49.902316 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:49.902293 5300 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-tmp-dir\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:20:49.902316 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:49.902316 5300 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-home\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:20:49.902457 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:49.902326 5300 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-model-cache\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:20:49.903806 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:49.903747 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-dshm" pod "ee55bf23-0c5c-4078-9149-ff5a6fab2c3d" (UID: "ee55bf23-0c5c-4078-9149-ff5a6fab2c3d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:20:49.903913 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:49.903862 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-tls-certs" pod "ee55bf23-0c5c-4078-9149-ff5a6fab2c3d" (UID: "ee55bf23-0c5c-4078-9149-ff5a6fab2c3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:20:49.904076 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:49.904057 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-kube-api-access-7vsnw" pod "ee55bf23-0c5c-4078-9149-ff5a6fab2c3d" (UID: "ee55bf23-0c5c-4078-9149-ff5a6fab2c3d"). InnerVolumeSpecName "kube-api-access-7vsnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:20:49.937402 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:49.937352 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:50.003805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:50.003679 5300 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-dshm\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:20:50.003805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:50.003713 5300 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-tls-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:20:50.003805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:50.003728 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vsnw\" (UniqueName: \"kubernetes.io/projected/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d-kube-api-access-7vsnw\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:20:50.561583 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:50.561544 5300 generic.go:356] "Generic (PLEG): container finished" podID="ee55bf23-0c5c-4078-9149-ff5a6fab2c3d" containerID="7fb2d755c9a5f4f70d52242ce2527ccfc17f70a648127975e4e1b8106746bc59" exitCode=0 Oct 30 01:20:50.562062 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:50.561604 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" Oct 30 01:20:50.562062 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:50.561636 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" event={"ID":"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d","Type":"ContainerDied","Data":"7fb2d755c9a5f4f70d52242ce2527ccfc17f70a648127975e4e1b8106746bc59"} Oct 30 01:20:50.562062 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:50.561671 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" event={"ID":"ee55bf23-0c5c-4078-9149-ff5a6fab2c3d","Type":"ContainerDied","Data":"105e5ac1cb91e62df879e0dc4122f6673f749e79f1fc80f38774dbd9ea625731"} Oct 30 01:20:50.562062 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:50.561691 5300 scope.go:122] "RemoveContainer" containerID="7fb2d755c9a5f4f70d52242ce2527ccfc17f70a648127975e4e1b8106746bc59" Oct 30 01:20:50.572860 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:50.572841 5300 scope.go:122] "RemoveContainer" containerID="7fb2d755c9a5f4f70d52242ce2527ccfc17f70a648127975e4e1b8106746bc59" Oct 30 01:20:50.573139 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:20:50.573119 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb2d755c9a5f4f70d52242ce2527ccfc17f70a648127975e4e1b8106746bc59\": container with ID starting with 7fb2d755c9a5f4f70d52242ce2527ccfc17f70a648127975e4e1b8106746bc59 not found: ID does not exist" containerID="7fb2d755c9a5f4f70d52242ce2527ccfc17f70a648127975e4e1b8106746bc59" Oct 30 01:20:50.573191 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:50.573148 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb2d755c9a5f4f70d52242ce2527ccfc17f70a648127975e4e1b8106746bc59"} err="failed to get container status \"7fb2d755c9a5f4f70d52242ce2527ccfc17f70a648127975e4e1b8106746bc59\": rpc error: code = NotFound desc = could not find container \"7fb2d755c9a5f4f70d52242ce2527ccfc17f70a648127975e4e1b8106746bc59\": container with ID starting with 7fb2d755c9a5f4f70d52242ce2527ccfc17f70a648127975e4e1b8106746bc59 not found: ID does not exist" Oct 30 01:20:50.618301 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:50.618262 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw"] Oct 30 01:20:50.631940 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:50.631908 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw"] Oct 30 01:20:50.717421 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:50.717382 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-eacea3/rate-test-model-eacea3-kserve-647f6df8b9-kt7jw" podUID="ee55bf23-0c5c-4078-9149-ff5a6fab2c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.129:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:20:51.604784 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:20:51.604695 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:20:51.680844 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:51.680807 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:51.823132 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:51.823099 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee55bf23-0c5c-4078-9149-ff5a6fab2c3d" path="/var/lib/kubelet/pods/ee55bf23-0c5c-4078-9149-ff5a6fab2c3d/volumes" Oct 30 01:20:52.188467 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:52.188426 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:52.289008 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:52.288976 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:53.408284 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:53.408255 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:55.156551 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:55.156494 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:55.999332 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:55.999287 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:56.432354 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:56.432271 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:56.614970 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:56.614937 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:56.921038 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:56.921004 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:57.008042 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:57.008010 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:57.481000 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:57.480967 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:58.262979 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:58.262949 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:20:59.979375 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:20:59.979335 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:00.085706 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.085673 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:00.834421 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.834318 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp"] Oct 30 01:21:00.835311 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.835288 5300 cpu_manager.go:414] "containerMap: removing container" podUID="ee55bf23-0c5c-4078-9149-ff5a6fab2c3d" containerName="main" Oct 30 01:21:00.835408 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.835314 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee55bf23-0c5c-4078-9149-ff5a6fab2c3d" containerName="main" Oct 30 01:21:00.835501 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.835488 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee55bf23-0c5c-4078-9149-ff5a6fab2c3d" containerName="main" Oct 30 01:21:00.840694 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.840673 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:00.866511 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.866474 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp"] Oct 30 01:21:00.872870 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.872838 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-e2d6d3-openshift-default-dockercfg-h487c\"" Oct 30 01:21:00.883532 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.883497 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-workload-socket\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:00.883686 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.883619 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1455aae2-7d1f-4788-9a94-7469bee17334-istio-token\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:00.883789 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.883745 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-credential-socket\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:00.884620 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.884591 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-workload-certs\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:00.885285 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.885262 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1455aae2-7d1f-4788-9a94-7469bee17334-istio-podinfo\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:00.885394 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.885348 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxzrs\" (UniqueName: \"kubernetes.io/projected/1455aae2-7d1f-4788-9a94-7469bee17334-kube-api-access-qxzrs\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:00.885458 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.885408 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-istio-envoy\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:00.885961 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.885931 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-istio-data\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:00.886064 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.885983 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1455aae2-7d1f-4788-9a94-7469bee17334-istiod-ca-cert\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:00.987728 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.987696 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-istio-data\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:00.988213 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.987735 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1455aae2-7d1f-4788-9a94-7469bee17334-istiod-ca-cert\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:00.988213 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.987753 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-workload-socket\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:00.988213 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.987804 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1455aae2-7d1f-4788-9a94-7469bee17334-istio-token\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:00.988213 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.987854 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-credential-socket\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:00.988213 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.987891 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-workload-certs\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:00.988213 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.988059 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1455aae2-7d1f-4788-9a94-7469bee17334-istio-podinfo\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:00.988213 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.988115 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxzrs\" (UniqueName: \"kubernetes.io/projected/1455aae2-7d1f-4788-9a94-7469bee17334-kube-api-access-qxzrs\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:00.988213 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.988170 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-istio-envoy\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:00.988663 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.988306 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-credential-socket\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:00.988663 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.988364 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-istio-data\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:00.988663 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.988403 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-workload-certs\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:00.988663 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.988451 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-workload-socket\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:00.989039 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.989016 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1455aae2-7d1f-4788-9a94-7469bee17334-istiod-ca-cert\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:00.990475 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.990453 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-istio-envoy\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:00.990604 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:00.990587 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1455aae2-7d1f-4788-9a94-7469bee17334-istio-podinfo\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:01.014811 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:01.014781 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1455aae2-7d1f-4788-9a94-7469bee17334-istio-token\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:01.014998 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:01.014983 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxzrs\" (UniqueName: \"kubernetes.io/projected/1455aae2-7d1f-4788-9a94-7469bee17334-kube-api-access-qxzrs\") pod \"e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:01.156297 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:01.156211 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:01.482181 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:01.482154 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp"] Oct 30 01:21:01.483109 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:21:01.483082 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1455aae2_7d1f_4788_9a94_7469bee17334.slice/crio-c06b8bc5785a4021677dbc806c370b192cfb2b07c5ca59474b6371261072ea71 WatchSource:0}: Error finding container c06b8bc5785a4021677dbc806c370b192cfb2b07c5ca59474b6371261072ea71: Status 404 returned error can't find the container with id c06b8bc5785a4021677dbc806c370b192cfb2b07c5ca59474b6371261072ea71 Oct 30 01:21:01.486042 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:01.486006 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:21:01.486157 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:01.486088 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:21:01.486157 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:01.486135 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:21:01.659733 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:01.659698 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" event={"ID":"1455aae2-7d1f-4788-9a94-7469bee17334","Type":"ContainerStarted","Data":"b6db75e823b319ca70be02011a1ea7b9a20ba9e92aeb44a5f3c935ad42adc3ee"} Oct 30 01:21:01.659967 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:01.659790 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" event={"ID":"1455aae2-7d1f-4788-9a94-7469bee17334","Type":"ContainerStarted","Data":"c06b8bc5785a4021677dbc806c370b192cfb2b07c5ca59474b6371261072ea71"} Oct 30 01:21:01.664639 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:01.664611 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:01.764110 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:01.764032 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:01.865345 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:01.865303 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:02.093789 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.093668 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" podStartSLOduration=2.093654017 podStartE2EDuration="2.093654017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:01.714147163 +0000 UTC m=+2262.030015493" watchObservedRunningTime="2026-10-30 01:21:02.093654017 +0000 UTC m=+2262.409522342" Oct 30 01:21:02.096203 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.096176 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl"] Oct 30 01:21:02.100866 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.100844 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.134140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.134112 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl"] Oct 30 01:21:02.145595 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.145563 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-886adb-openshift-default-dockercfg-l6rhb\"" Oct 30 01:21:02.156825 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.156792 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:02.158337 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.158302 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 01:21:02.158544 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.158375 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" podUID="1455aae2-7d1f-4788-9a94-7469bee17334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 01:21:02.211986 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.211950 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.212122 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.212071 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/508889de-4033-41dd-9b1a-88925f799310-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.212261 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.212241 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.212315 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.212276 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.212410 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.212392 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/508889de-4033-41dd-9b1a-88925f799310-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.212447 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.212434 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.212548 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.212534 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-746b9\" (UniqueName: \"kubernetes.io/projected/508889de-4033-41dd-9b1a-88925f799310-kube-api-access-746b9\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.212585 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.212566 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.212646 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.212634 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/508889de-4033-41dd-9b1a-88925f799310-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.314465 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.314431 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.314657 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.314505 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/508889de-4033-41dd-9b1a-88925f799310-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.314657 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.314637 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.314814 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.314680 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.314814 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.314731 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/508889de-4033-41dd-9b1a-88925f799310-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.314814 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.314810 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.314993 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.314890 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-746b9\" (UniqueName: \"kubernetes.io/projected/508889de-4033-41dd-9b1a-88925f799310-kube-api-access-746b9\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.314993 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.314920 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.314993 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.314923 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.315189 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.315007 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/508889de-4033-41dd-9b1a-88925f799310-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.315189 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.315054 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.315300 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.315245 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.315485 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.315467 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/508889de-4033-41dd-9b1a-88925f799310-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.315599 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.315576 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.317255 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.317235 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.317599 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.317575 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/508889de-4033-41dd-9b1a-88925f799310-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.339275 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.339242 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/508889de-4033-41dd-9b1a-88925f799310-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.339425 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.339279 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-746b9\" (UniqueName: \"kubernetes.io/projected/508889de-4033-41dd-9b1a-88925f799310-kube-api-access-746b9\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.424850 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.424130 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:02.759871 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.759842 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl"] Oct 30 01:21:02.760033 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:21:02.760012 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508889de_4033_41dd_9b1a_88925f799310.slice/crio-a03cae68de7307249d2779de88c448359893897d4375d7a222997dd3e187992e WatchSource:0}: Error finding container a03cae68de7307249d2779de88c448359893897d4375d7a222997dd3e187992e: Status 404 returned error can't find the container with id a03cae68de7307249d2779de88c448359893897d4375d7a222997dd3e187992e Oct 30 01:21:02.762964 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.762929 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:21:02.763061 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.763003 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:21:02.763061 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:02.763039 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:21:03.157465 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:03.157368 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 01:21:03.157984 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:03.157455 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" podUID="1455aae2-7d1f-4788-9a94-7469bee17334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 01:21:03.554135 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:03.554105 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:03.683511 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:03.683475 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" event={"ID":"508889de-4033-41dd-9b1a-88925f799310","Type":"ContainerStarted","Data":"c2b85899622be1e89574dfb021893a6f5c66d0fc7a6167fcd129ac09aa300a60"} Oct 30 01:21:03.683782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:03.683740 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" event={"ID":"508889de-4033-41dd-9b1a-88925f799310","Type":"ContainerStarted","Data":"a03cae68de7307249d2779de88c448359893897d4375d7a222997dd3e187992e"} Oct 30 01:21:04.159785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:04.159208 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 01:21:04.159785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:04.159276 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" podUID="1455aae2-7d1f-4788-9a94-7469bee17334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 01:21:04.429843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:04.427796 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:04.429843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:04.427941 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 01:21:04.429843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:04.427996 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" podUID="508889de-4033-41dd-9b1a-88925f799310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 01:21:04.486533 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:04.486490 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:04.967783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:04.967436 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:05.168838 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:05.168481 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 01:21:05.168838 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:05.168561 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" podUID="1455aae2-7d1f-4788-9a94-7469bee17334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 01:21:05.415494 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:05.415464 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:05.426325 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:05.426294 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 01:21:05.434818 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:05.426554 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" podUID="508889de-4033-41dd-9b1a-88925f799310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 01:21:05.559687 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:05.554253 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:06.161788 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:06.160125 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 01:21:06.161788 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:06.160186 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" podUID="1455aae2-7d1f-4788-9a94-7469bee17334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 01:21:06.438842 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:06.435167 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 01:21:06.438842 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:06.435230 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" podUID="508889de-4033-41dd-9b1a-88925f799310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 01:21:06.461340 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:06.461181 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:07.157203 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:07.157161 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 01:21:07.157490 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:07.157458 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" podUID="1455aae2-7d1f-4788-9a94-7469bee17334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 01:21:07.368447 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:07.368400 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:07.425166 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:07.425080 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 01:21:07.425418 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:07.425388 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" podUID="508889de-4033-41dd-9b1a-88925f799310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 01:21:08.215806 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.215739 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" podStartSLOduration=6.215722868 podStartE2EDuration="6.215722868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:03.775279253 +0000 UTC m=+2264.091147579" watchObservedRunningTime="2026-10-30 01:21:08.215722868 +0000 UTC m=+2268.531591193" Oct 30 01:21:08.216507 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.216483 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2"] Oct 30 01:21:08.261814 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.261092 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" Oct 30 01:21:08.268818 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.266861 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2"] Oct 30 01:21:08.275283 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.274166 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:08.291779 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.291734 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-e4f8a6\"/\"openshift-service-ca.crt\"" Oct 30 01:21:08.291956 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.291878 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e4f8a6\"/\"default-dockercfg-p2gcj\"" Oct 30 01:21:08.291956 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.291888 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e4f8a6\"/\"sub-test-model-e4f8a6-kserve-self-signed-certs\"" Oct 30 01:21:08.292167 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.292141 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-e4f8a6\"/\"kube-root-ca.crt\"" Oct 30 01:21:08.295799 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.295773 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-model-cache\") pod \"sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2\" (UID: \"ff735ea9-e315-46b8-b613-8164e6a2823a\") " pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" Oct 30 01:21:08.295925 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.295897 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-home\") pod \"sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2\" (UID: \"ff735ea9-e315-46b8-b613-8164e6a2823a\") " pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" Oct 30 01:21:08.296040 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.296020 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-dshm\") pod \"sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2\" (UID: \"ff735ea9-e315-46b8-b613-8164e6a2823a\") " pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" Oct 30 01:21:08.296101 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.296074 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-tmp-dir\") pod \"sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2\" (UID: \"ff735ea9-e315-46b8-b613-8164e6a2823a\") " pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" Oct 30 01:21:08.296171 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.296158 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff735ea9-e315-46b8-b613-8164e6a2823a-tls-certs\") pod \"sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2\" (UID: \"ff735ea9-e315-46b8-b613-8164e6a2823a\") " pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" Oct 30 01:21:08.296248 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.296230 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpl7m\" (UniqueName: \"kubernetes.io/projected/ff735ea9-e315-46b8-b613-8164e6a2823a-kube-api-access-xpl7m\") pod \"sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2\" (UID: \"ff735ea9-e315-46b8-b613-8164e6a2823a\") " pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" Oct 30 01:21:08.309387 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.309363 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:08.311062 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.311045 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:08.312050 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.312037 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:21:08.397713 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.397680 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-home\") pod \"sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2\" (UID: \"ff735ea9-e315-46b8-b613-8164e6a2823a\") " pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" Oct 30 01:21:08.397948 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.397747 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-dshm\") pod \"sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2\" (UID: \"ff735ea9-e315-46b8-b613-8164e6a2823a\") " pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" Oct 30 01:21:08.397948 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.397831 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-tmp-dir\") pod \"sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2\" (UID: \"ff735ea9-e315-46b8-b613-8164e6a2823a\") " pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" Oct 30 01:21:08.397948 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.397885 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff735ea9-e315-46b8-b613-8164e6a2823a-tls-certs\") pod \"sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2\" (UID: \"ff735ea9-e315-46b8-b613-8164e6a2823a\") " pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" Oct 30 01:21:08.397948 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.397940 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpl7m\" (UniqueName: \"kubernetes.io/projected/ff735ea9-e315-46b8-b613-8164e6a2823a-kube-api-access-xpl7m\") pod \"sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2\" (UID: \"ff735ea9-e315-46b8-b613-8164e6a2823a\") " pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" Oct 30 01:21:08.398184 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.398164 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-home\") pod \"sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2\" (UID: \"ff735ea9-e315-46b8-b613-8164e6a2823a\") " pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" Oct 30 01:21:08.398253 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.398190 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-tmp-dir\") pod \"sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2\" (UID: \"ff735ea9-e315-46b8-b613-8164e6a2823a\") " pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" Oct 30 01:21:08.398253 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.398169 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-model-cache\") pod \"sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2\" (UID: \"ff735ea9-e315-46b8-b613-8164e6a2823a\") " pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" Oct 30 01:21:08.398677 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.398648 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-model-cache\") pod \"sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2\" (UID: \"ff735ea9-e315-46b8-b613-8164e6a2823a\") " pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" Oct 30 01:21:08.400907 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.400881 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff735ea9-e315-46b8-b613-8164e6a2823a-tls-certs\") pod \"sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2\" (UID: \"ff735ea9-e315-46b8-b613-8164e6a2823a\") " pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" Oct 30 01:21:08.401053 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.400891 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-dshm\") pod \"sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2\" (UID: \"ff735ea9-e315-46b8-b613-8164e6a2823a\") " pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" Oct 30 01:21:08.451103 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.449365 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 01:21:08.451103 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.450698 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" podUID="508889de-4033-41dd-9b1a-88925f799310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 01:21:08.459997 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.459933 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpl7m\" (UniqueName: \"kubernetes.io/projected/ff735ea9-e315-46b8-b613-8164e6a2823a-kube-api-access-xpl7m\") pod \"sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2\" (UID: \"ff735ea9-e315-46b8-b613-8164e6a2823a\") " pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" Oct 30 01:21:08.481040 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.481005 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5"] Oct 30 01:21:08.487494 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.487463 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.502259 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.502229 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5"] Oct 30 01:21:08.578881 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.578819 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-2be695-openshift-default-dockercfg-f5bng\"" Oct 30 01:21:08.579208 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.579183 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" Oct 30 01:21:08.604709 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.604116 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.604709 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.604189 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtthd\" (UniqueName: \"kubernetes.io/projected/653f0ddb-957e-4dae-85c9-b5822892df36-kube-api-access-mtthd\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.604709 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.604228 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.604709 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.604334 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.604709 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.604439 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/653f0ddb-957e-4dae-85c9-b5822892df36-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.604709 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.604479 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.604709 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.604555 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/653f0ddb-957e-4dae-85c9-b5822892df36-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.604709 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.604625 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/653f0ddb-957e-4dae-85c9-b5822892df36-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.606046 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.605995 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.709336 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.708153 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/653f0ddb-957e-4dae-85c9-b5822892df36-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.709336 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.708223 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.709336 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.708307 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.709336 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.708362 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtthd\" (UniqueName: \"kubernetes.io/projected/653f0ddb-957e-4dae-85c9-b5822892df36-kube-api-access-mtthd\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.709336 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.708396 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.709336 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.708494 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.709336 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.708593 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/653f0ddb-957e-4dae-85c9-b5822892df36-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.709336 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.708633 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.709336 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.708710 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/653f0ddb-957e-4dae-85c9-b5822892df36-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.709846 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.709828 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/653f0ddb-957e-4dae-85c9-b5822892df36-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.710691 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.710660 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.710924 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.710903 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.711151 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.711133 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.711942 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.711916 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.762674 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.762585 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/653f0ddb-957e-4dae-85c9-b5822892df36-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.769858 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.769821 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtthd\" (UniqueName: \"kubernetes.io/projected/653f0ddb-957e-4dae-85c9-b5822892df36-kube-api-access-mtthd\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.788006 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.787557 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.790591 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.790528 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/653f0ddb-957e-4dae-85c9-b5822892df36-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.814267 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.814021 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:08.960945 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:08.958084 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2"] Oct 30 01:21:09.156443 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:21:09.156415 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653f0ddb_957e_4dae_85c9_b5822892df36.slice/crio-cae8043b39ee4bf4a486b9ecc04f2d25ddc89f72d015a86b0af35fd2dbe43013 WatchSource:0}: Error finding container cae8043b39ee4bf4a486b9ecc04f2d25ddc89f72d015a86b0af35fd2dbe43013: Status 404 returned error can't find the container with id cae8043b39ee4bf4a486b9ecc04f2d25ddc89f72d015a86b0af35fd2dbe43013 Oct 30 01:21:09.158437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:09.158413 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5"] Oct 30 01:21:09.158888 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:09.158853 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:21:09.158991 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:09.158923 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:21:09.158991 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:09.158951 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:21:09.333697 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:09.333551 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:09.460682 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:09.460648 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:09.461423 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:09.461396 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:09.462438 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:09.462422 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:21:09.797975 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:09.797933 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" event={"ID":"653f0ddb-957e-4dae-85c9-b5822892df36","Type":"ContainerStarted","Data":"606ebe0aff4314afb463b0fc678e264041553c36c6bef8ae5bc95637b2cabbdb"} Oct 30 01:21:09.798182 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:09.798000 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" event={"ID":"653f0ddb-957e-4dae-85c9-b5822892df36","Type":"ContainerStarted","Data":"cae8043b39ee4bf4a486b9ecc04f2d25ddc89f72d015a86b0af35fd2dbe43013"} Oct 30 01:21:09.799796 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:09.799733 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" event={"ID":"ff735ea9-e315-46b8-b613-8164e6a2823a","Type":"ContainerStarted","Data":"bbed55714c76a1608f6259095b6d39a19dec6628797f808311c821acc2d00e37"} Oct 30 01:21:09.799925 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:09.799812 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" event={"ID":"ff735ea9-e315-46b8-b613-8164e6a2823a","Type":"ContainerStarted","Data":"908222fb45fb16e83a3d5d6e9b6e2a660364e9f6d80ff849f1103b56fff769ed"} Oct 30 01:21:09.863957 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:09.863925 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:09.866345 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:09.866249 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" podStartSLOduration=1.8662350989999998 podStartE2EDuration="1.866235099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:09.865446954 +0000 UTC m=+2270.181315279" watchObservedRunningTime="2026-10-30 01:21:09.866235099 +0000 UTC m=+2270.182103423" Oct 30 01:21:09.907809 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:09.907736 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" podStartSLOduration=1.907721269 podStartE2EDuration="1.907721269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:09.907041883 +0000 UTC m=+2270.222910207" watchObservedRunningTime="2026-10-30 01:21:09.907721269 +0000 UTC m=+2270.223589594" Oct 30 01:21:10.579740 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:10.579695 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" Oct 30 01:21:10.817395 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:10.817360 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 01:21:10.817574 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:10.817416 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" podUID="653f0ddb-957e-4dae-85c9-b5822892df36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 01:21:10.954241 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:10.954151 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:11.815209 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:11.815171 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 01:21:11.815655 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:11.815235 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" podUID="653f0ddb-957e-4dae-85c9-b5822892df36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 01:21:12.575554 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:12.575511 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:12.814576 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:12.814542 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 01:21:12.814747 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:12.814605 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" podUID="653f0ddb-957e-4dae-85c9-b5822892df36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 01:21:13.814797 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:13.814725 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 01:21:13.815392 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:13.815360 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" podUID="653f0ddb-957e-4dae-85c9-b5822892df36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 01:21:14.413112 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:14.413072 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:14.815146 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:14.815115 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 30 01:21:14.815548 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:14.815162 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" podUID="653f0ddb-957e-4dae-85c9-b5822892df36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 30 01:21:15.905671 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:15.905637 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:15.906176 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:15.906148 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:15.907343 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:15.907319 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:21:16.009388 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:16.009359 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:16.163019 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:16.162938 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:16.267009 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:16.266972 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:17.232912 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:17.232859 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl"] Oct 30 01:21:17.282273 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:17.282178 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:17.411656 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:17.411620 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:17.531776 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:17.531668 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:17.917824 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:17.917727 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:18.580428 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:18.580389 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" Oct 30 01:21:18.593914 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:18.593882 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" Oct 30 01:21:18.937785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:18.937668 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j"] Oct 30 01:21:18.948982 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:18.948953 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" Oct 30 01:21:18.959368 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:18.959336 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:19.532200 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:19.532166 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:19.778092 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:19.778058 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:20.178823 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:20.178755 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:20.669327 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:20.669299 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:21.321265 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:21.321213 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:21.423941 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:21.423909 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:21.540285 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:21.540251 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt"] Oct 30 01:21:21.546058 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:21.546037 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt" Oct 30 01:21:21.570845 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:21.570816 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt"] Oct 30 01:21:21.579869 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:21.579787 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-e92648\"" Oct 30 01:21:21.579869 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:21.579815 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-e92648-dockercfg-cwhjd\"" Oct 30 01:21:21.602243 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:21:21.602204 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:21:21.641201 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:21.641164 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh"] Oct 30 01:21:21.646506 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:21.646474 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh" Oct 30 01:21:21.716003 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:21.715961 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b27lj\" (UniqueName: \"kubernetes.io/projected/19df0d00-62d5-4c6c-9cba-673cd02dbb8a-kube-api-access-b27lj\") pod \"payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt\" (UID: \"19df0d00-62d5-4c6c-9cba-673cd02dbb8a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt" Oct 30 01:21:21.716328 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:21.716302 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19df0d00-62d5-4c6c-9cba-673cd02dbb8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt\" (UID: \"19df0d00-62d5-4c6c-9cba-673cd02dbb8a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt" Oct 30 01:21:21.722782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:21.717210 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh"] Oct 30 01:21:21.817725 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:21.817685 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgntw\" (UniqueName: \"kubernetes.io/projected/aa7f5966-fe14-4c2c-af88-ac82100209ba-kube-api-access-dgntw\") pod \"payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh\" (UID: \"aa7f5966-fe14-4c2c-af88-ac82100209ba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh" Oct 30 01:21:21.817941 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:21.817867 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b27lj\" (UniqueName: \"kubernetes.io/projected/19df0d00-62d5-4c6c-9cba-673cd02dbb8a-kube-api-access-b27lj\") pod \"payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt\" (UID: \"19df0d00-62d5-4c6c-9cba-673cd02dbb8a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt" Oct 30 01:21:21.817941 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:21.817898 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa7f5966-fe14-4c2c-af88-ac82100209ba-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh\" (UID: \"aa7f5966-fe14-4c2c-af88-ac82100209ba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh" Oct 30 01:21:21.818067 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:21.817950 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19df0d00-62d5-4c6c-9cba-673cd02dbb8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt\" (UID: \"19df0d00-62d5-4c6c-9cba-673cd02dbb8a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt" Oct 30 01:21:21.818684 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:21.818660 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19df0d00-62d5-4c6c-9cba-673cd02dbb8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt\" (UID: \"19df0d00-62d5-4c6c-9cba-673cd02dbb8a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt" Oct 30 01:21:21.840794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:21.840722 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b27lj\" (UniqueName: \"kubernetes.io/projected/19df0d00-62d5-4c6c-9cba-673cd02dbb8a-kube-api-access-b27lj\") pod \"payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt\" (UID: \"19df0d00-62d5-4c6c-9cba-673cd02dbb8a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt" Oct 30 01:21:21.860439 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:21.860412 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt" Oct 30 01:21:21.919617 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:21.919570 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgntw\" (UniqueName: \"kubernetes.io/projected/aa7f5966-fe14-4c2c-af88-ac82100209ba-kube-api-access-dgntw\") pod \"payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh\" (UID: \"aa7f5966-fe14-4c2c-af88-ac82100209ba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh" Oct 30 01:21:21.919863 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:21.919730 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa7f5966-fe14-4c2c-af88-ac82100209ba-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh\" (UID: \"aa7f5966-fe14-4c2c-af88-ac82100209ba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh" Oct 30 01:21:21.920693 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:21.920662 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa7f5966-fe14-4c2c-af88-ac82100209ba-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh\" (UID: \"aa7f5966-fe14-4c2c-af88-ac82100209ba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh" Oct 30 01:21:21.953225 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:21.952213 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgntw\" (UniqueName: \"kubernetes.io/projected/aa7f5966-fe14-4c2c-af88-ac82100209ba-kube-api-access-dgntw\") pod \"payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh\" (UID: \"aa7f5966-fe14-4c2c-af88-ac82100209ba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh" Oct 30 01:21:22.100698 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:22.100616 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh" Oct 30 01:21:22.167748 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:22.167718 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt"] Oct 30 01:21:22.168119 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:21:22.168093 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19df0d00_62d5_4c6c_9cba_673cd02dbb8a.slice/crio-2467892c11193f03857f979ca0627612f097022297fa3f203f25e193ad069c3e WatchSource:0}: Error finding container 2467892c11193f03857f979ca0627612f097022297fa3f203f25e193ad069c3e: Status 404 returned error can't find the container with id 2467892c11193f03857f979ca0627612f097022297fa3f203f25e193ad069c3e Oct 30 01:21:22.413817 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:22.413788 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh"] Oct 30 01:21:22.414211 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:21:22.414102 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa7f5966_fe14_4c2c_af88_ac82100209ba.slice/crio-bb33d4402f770124924258fc4ea02492930d59488e4dd4de75e49ba2429c0ebc WatchSource:0}: Error finding container bb33d4402f770124924258fc4ea02492930d59488e4dd4de75e49ba2429c0ebc: Status 404 returned error can't find the container with id bb33d4402f770124924258fc4ea02492930d59488e4dd4de75e49ba2429c0ebc Oct 30 01:21:22.970862 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:22.970813 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh" event={"ID":"aa7f5966-fe14-4c2c-af88-ac82100209ba","Type":"ContainerStarted","Data":"0cd561fc955ec100b8d6ddcd2a858e9c4c5004e428c9235caa8e915b013486c7"} Oct 30 01:21:22.971067 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:22.970877 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh" event={"ID":"aa7f5966-fe14-4c2c-af88-ac82100209ba","Type":"ContainerStarted","Data":"bb33d4402f770124924258fc4ea02492930d59488e4dd4de75e49ba2429c0ebc"} Oct 30 01:21:22.971145 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:22.971089 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh" Oct 30 01:21:22.973363 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:22.973321 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt" event={"ID":"19df0d00-62d5-4c6c-9cba-673cd02dbb8a","Type":"ContainerStarted","Data":"3d1bc8568f9354600975d6e738079c2dfba0e59d31b12455e7ed2b14bd1e2b6c"} Oct 30 01:21:22.973515 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:22.973378 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt" event={"ID":"19df0d00-62d5-4c6c-9cba-673cd02dbb8a","Type":"ContainerStarted","Data":"2467892c11193f03857f979ca0627612f097022297fa3f203f25e193ad069c3e"} Oct 30 01:21:22.973515 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:22.973411 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt" Oct 30 01:21:23.018122 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:23.018070 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh" podStartSLOduration=2.01805514 podStartE2EDuration="2.01805514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:23.016144539 +0000 UTC m=+2283.332012865" watchObservedRunningTime="2026-10-30 01:21:23.01805514 +0000 UTC m=+2283.333923477" Oct 30 01:21:23.019982 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:23.019958 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:23.059116 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:23.059065 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt" podStartSLOduration=2.05904935 podStartE2EDuration="2.05904935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:23.05780283 +0000 UTC m=+2283.373671169" watchObservedRunningTime="2026-10-30 01:21:23.05904935 +0000 UTC m=+2283.374917674" Oct 30 01:21:24.715649 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:24.715605 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:24.872936 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:24.872896 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:24.988585 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:24.988539 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:26.702597 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:26.702554 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:26.796473 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:26.796443 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:26.954960 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:26.954871 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr"] Oct 30 01:21:26.972151 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:26.972120 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr" Oct 30 01:21:26.976411 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:26.976379 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr"] Oct 30 01:21:26.988651 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:26.988624 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4be164\"" Oct 30 01:21:27.013781 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:27.002984 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4be164-dockercfg-f42nf\"" Oct 30 01:21:27.121669 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:27.121620 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gln9\" (UniqueName: \"kubernetes.io/projected/adffe4ff-92a0-46de-93f0-e1f0a46da05d-kube-api-access-9gln9\") pod \"payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr\" (UID: \"adffe4ff-92a0-46de-93f0-e1f0a46da05d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr" Oct 30 01:21:27.121903 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:27.121876 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adffe4ff-92a0-46de-93f0-e1f0a46da05d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr\" (UID: \"adffe4ff-92a0-46de-93f0-e1f0a46da05d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr" Oct 30 01:21:27.223253 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:27.223219 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adffe4ff-92a0-46de-93f0-e1f0a46da05d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr\" (UID: \"adffe4ff-92a0-46de-93f0-e1f0a46da05d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr" Oct 30 01:21:27.223433 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:27.223412 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gln9\" (UniqueName: \"kubernetes.io/projected/adffe4ff-92a0-46de-93f0-e1f0a46da05d-kube-api-access-9gln9\") pod \"payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr\" (UID: \"adffe4ff-92a0-46de-93f0-e1f0a46da05d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr" Oct 30 01:21:27.224084 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:27.224059 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adffe4ff-92a0-46de-93f0-e1f0a46da05d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr\" (UID: \"adffe4ff-92a0-46de-93f0-e1f0a46da05d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr" Oct 30 01:21:27.244744 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:27.244710 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gln9\" (UniqueName: \"kubernetes.io/projected/adffe4ff-92a0-46de-93f0-e1f0a46da05d-kube-api-access-9gln9\") pod \"payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr\" (UID: \"adffe4ff-92a0-46de-93f0-e1f0a46da05d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr" Oct 30 01:21:27.296100 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:27.296072 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr" Oct 30 01:21:27.605372 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:27.605340 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr"] Oct 30 01:21:27.606220 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:21:27.606195 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadffe4ff_92a0_46de_93f0_e1f0a46da05d.slice/crio-cf59edb0c26480f1465af96c9bdc60a5cb7542e89ff08cee71a9d4f9dfa97734 WatchSource:0}: Error finding container cf59edb0c26480f1465af96c9bdc60a5cb7542e89ff08cee71a9d4f9dfa97734: Status 404 returned error can't find the container with id cf59edb0c26480f1465af96c9bdc60a5cb7542e89ff08cee71a9d4f9dfa97734 Oct 30 01:21:28.030453 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:28.030414 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr" event={"ID":"adffe4ff-92a0-46de-93f0-e1f0a46da05d","Type":"ContainerStarted","Data":"31d6f878038a28754160021e18135b16ca02d2f109676353bbe5748f80cd8227"} Oct 30 01:21:28.030867 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:28.030635 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr" Oct 30 01:21:28.030867 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:28.030668 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr" event={"ID":"adffe4ff-92a0-46de-93f0-e1f0a46da05d","Type":"ContainerStarted","Data":"cf59edb0c26480f1465af96c9bdc60a5cb7542e89ff08cee71a9d4f9dfa97734"} Oct 30 01:21:28.074931 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:28.074870 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr" podStartSLOduration=2.074856038 podStartE2EDuration="2.074856038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:28.073973781 +0000 UTC m=+2288.389842118" watchObservedRunningTime="2026-10-30 01:21:28.074856038 +0000 UTC m=+2288.390724364" Oct 30 01:21:28.245251 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:28.245201 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:29.744416 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:29.744377 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:29.931021 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:29.930990 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:30.031966 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:30.031884 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:31.453961 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:31.453925 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:32.288305 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:32.288245 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" podUID="b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7" containerName="main" containerID="cri-o://f90d40a21b8ef4968166db4c1d1c637dfc0000d174285abed9c0062b7bfddf67" gracePeriod=45 Oct 30 01:21:32.594862 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:32.594835 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" Oct 30 01:21:32.710690 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:32.710647 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-tmp-dir\") pod \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\" (UID: \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\") " Oct 30 01:21:32.710910 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:32.710716 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-kube-api-access-qt6hl\" (UniqueName: \"kubernetes.io/projected/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-kube-api-access-qt6hl\") pod \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\" (UID: \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\") " Oct 30 01:21:32.710910 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:32.710789 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-tls-certs\") pod \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\" (UID: \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\") " Oct 30 01:21:32.710910 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:32.710874 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-home\" (UniqueName: \"kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-home\") pod \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\" (UID: \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\") " Oct 30 01:21:32.711107 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:32.710968 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-dshm\") pod \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\" (UID: \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\") " Oct 30 01:21:32.711107 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:32.711003 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-tmp-dir" pod "b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7" (UID: "b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:32.711107 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:32.711048 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-model-cache\") pod \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\" (UID: \"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7\") " Oct 30 01:21:32.711287 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:32.711166 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-home" pod "b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7" (UID: "b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:32.711346 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:32.711319 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-model-cache" pod "b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7" (UID: "b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:32.711861 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:32.711832 5300 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-model-cache\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:21:32.711979 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:32.711888 5300 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-tmp-dir\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:21:32.711979 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:32.711903 5300 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-home\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:21:32.713209 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:32.713183 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-tls-certs" pod "b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7" (UID: "b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:21:32.713321 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:32.713186 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-dshm" pod "b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7" (UID: "b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:32.713321 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:32.713248 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-kube-api-access-qt6hl" pod "b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7" (UID: "b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7"). InnerVolumeSpecName "kube-api-access-qt6hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:32.813742 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:32.813655 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt6hl\" (UniqueName: \"kubernetes.io/projected/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-kube-api-access-qt6hl\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:21:32.813742 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:32.813697 5300 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-tls-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:21:32.813742 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:32.813711 5300 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7-dshm\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:21:32.914075 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:32.914042 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:33.074440 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:33.074351 5300 generic.go:356] "Generic (PLEG): container finished" podID="b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7" containerID="f90d40a21b8ef4968166db4c1d1c637dfc0000d174285abed9c0062b7bfddf67" exitCode=0 Oct 30 01:21:33.074686 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:33.074465 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" event={"ID":"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7","Type":"ContainerDied","Data":"f90d40a21b8ef4968166db4c1d1c637dfc0000d174285abed9c0062b7bfddf67"} Oct 30 01:21:33.074686 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:33.074492 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" event={"ID":"b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7","Type":"ContainerDied","Data":"24a951f3066f0b5613b055ff82622414737054e9ea4653068eaf2928d73c4886"} Oct 30 01:21:33.074686 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:33.074507 5300 scope.go:122] "RemoveContainer" containerID="f90d40a21b8ef4968166db4c1d1c637dfc0000d174285abed9c0062b7bfddf67" Oct 30 01:21:33.074686 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:33.074640 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl" Oct 30 01:21:33.092872 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:33.092551 5300 scope.go:122] "RemoveContainer" containerID="f90d40a21b8ef4968166db4c1d1c637dfc0000d174285abed9c0062b7bfddf67" Oct 30 01:21:33.093008 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:21:33.092984 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f90d40a21b8ef4968166db4c1d1c637dfc0000d174285abed9c0062b7bfddf67\": container with ID starting with f90d40a21b8ef4968166db4c1d1c637dfc0000d174285abed9c0062b7bfddf67 not found: ID does not exist" containerID="f90d40a21b8ef4968166db4c1d1c637dfc0000d174285abed9c0062b7bfddf67" Oct 30 01:21:33.093070 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:33.093017 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f90d40a21b8ef4968166db4c1d1c637dfc0000d174285abed9c0062b7bfddf67"} err="failed to get container status \"f90d40a21b8ef4968166db4c1d1c637dfc0000d174285abed9c0062b7bfddf67\": rpc error: code = NotFound desc = could not find container \"f90d40a21b8ef4968166db4c1d1c637dfc0000d174285abed9c0062b7bfddf67\": container with ID starting with f90d40a21b8ef4968166db4c1d1c637dfc0000d174285abed9c0062b7bfddf67 not found: ID does not exist" Oct 30 01:21:33.133964 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:33.133929 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl"] Oct 30 01:21:33.147614 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:33.147584 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-64f472/sub-test-model-64f472-kserve-b5488fb95-xkbfl"] Oct 30 01:21:33.824212 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:33.824182 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7" path="/var/lib/kubelet/pods/b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7/volumes" Oct 30 01:21:33.960686 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:33.960633 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" podUID="31ba3725-cc95-463d-9580-513e3a0354c0" containerName="main" containerID="cri-o://72ba100c0106b7f72b67645e16d273975b8a0efcd482dd78f2a35a0b448580c7" gracePeriod=45 Oct 30 01:21:33.984472 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:33.984443 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh" Oct 30 01:21:33.984621 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:33.984606 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt" Oct 30 01:21:34.084203 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:34.084150 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:34.086218 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:34.086193 5300 generic.go:356] "Generic (PLEG): container finished" podID="31ba3725-cc95-463d-9580-513e3a0354c0" containerID="72ba100c0106b7f72b67645e16d273975b8a0efcd482dd78f2a35a0b448580c7" exitCode=0 Oct 30 01:21:34.086330 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:34.086267 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" event={"ID":"31ba3725-cc95-463d-9580-513e3a0354c0","Type":"ContainerDied","Data":"72ba100c0106b7f72b67645e16d273975b8a0efcd482dd78f2a35a0b448580c7"} Oct 30 01:21:34.187629 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:34.187581 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:34.265421 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:34.265389 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" Oct 30 01:21:34.334167 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:34.334077 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31ba3725-cc95-463d-9580-513e3a0354c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/31ba3725-cc95-463d-9580-513e3a0354c0-tls-certs\") pod \"31ba3725-cc95-463d-9580-513e3a0354c0\" (UID: \"31ba3725-cc95-463d-9580-513e3a0354c0\") " Oct 30 01:21:34.334349 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:34.334200 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-dshm\") pod \"31ba3725-cc95-463d-9580-513e3a0354c0\" (UID: \"31ba3725-cc95-463d-9580-513e3a0354c0\") " Oct 30 01:21:34.334349 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:34.334234 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31ba3725-cc95-463d-9580-513e3a0354c0-kube-api-access-hwdw4\" (UniqueName: \"kubernetes.io/projected/31ba3725-cc95-463d-9580-513e3a0354c0-kube-api-access-hwdw4\") pod \"31ba3725-cc95-463d-9580-513e3a0354c0\" (UID: \"31ba3725-cc95-463d-9580-513e3a0354c0\") " Oct 30 01:21:34.334349 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:34.334303 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-model-cache\") pod \"31ba3725-cc95-463d-9580-513e3a0354c0\" (UID: \"31ba3725-cc95-463d-9580-513e3a0354c0\") " Oct 30 01:21:34.334542 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:34.334366 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-home\" (UniqueName: \"kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-home\") pod \"31ba3725-cc95-463d-9580-513e3a0354c0\" (UID: \"31ba3725-cc95-463d-9580-513e3a0354c0\") " Oct 30 01:21:34.334542 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:34.334401 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-tmp-dir\") pod \"31ba3725-cc95-463d-9580-513e3a0354c0\" (UID: \"31ba3725-cc95-463d-9580-513e3a0354c0\") " Oct 30 01:21:34.334698 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:34.334661 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-model-cache" pod "31ba3725-cc95-463d-9580-513e3a0354c0" (UID: "31ba3725-cc95-463d-9580-513e3a0354c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:34.334999 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:34.334750 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-home" pod "31ba3725-cc95-463d-9580-513e3a0354c0" (UID: "31ba3725-cc95-463d-9580-513e3a0354c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:34.334999 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:34.334944 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-tmp-dir" pod "31ba3725-cc95-463d-9580-513e3a0354c0" (UID: "31ba3725-cc95-463d-9580-513e3a0354c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:34.335225 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:34.335203 5300 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-model-cache\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:21:34.335308 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:34.335229 5300 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-home\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:21:34.335308 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:34.335242 5300 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-tmp-dir\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:21:34.337019 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:34.336982 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ba3725-cc95-463d-9580-513e3a0354c0-kube-api-access-hwdw4" pod "31ba3725-cc95-463d-9580-513e3a0354c0" (UID: "31ba3725-cc95-463d-9580-513e3a0354c0"). InnerVolumeSpecName "kube-api-access-hwdw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:34.337167 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:34.337143 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ba3725-cc95-463d-9580-513e3a0354c0-tls-certs" pod "31ba3725-cc95-463d-9580-513e3a0354c0" (UID: "31ba3725-cc95-463d-9580-513e3a0354c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:21:34.337479 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:34.337444 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-dshm" pod "31ba3725-cc95-463d-9580-513e3a0354c0" (UID: "31ba3725-cc95-463d-9580-513e3a0354c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:34.353735 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:34.353607 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:34.436628 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:34.436577 5300 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ba3725-cc95-463d-9580-513e3a0354c0-tls-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:21:34.436628 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:34.436612 5300 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ba3725-cc95-463d-9580-513e3a0354c0-dshm\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:21:34.436628 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:34.436622 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwdw4\" (UniqueName: \"kubernetes.io/projected/31ba3725-cc95-463d-9580-513e3a0354c0-kube-api-access-hwdw4\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:21:35.097321 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:35.097283 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" Oct 30 01:21:35.097321 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:35.097318 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" event={"ID":"31ba3725-cc95-463d-9580-513e3a0354c0","Type":"ContainerDied","Data":"6bd699c1a86815bdc89a56bc49c51654f309e41b687c2c7866934183f4d33b1b"} Oct 30 01:21:35.097847 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:35.097368 5300 scope.go:122] "RemoveContainer" containerID="72ba100c0106b7f72b67645e16d273975b8a0efcd482dd78f2a35a0b448580c7" Oct 30 01:21:35.160164 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:35.160131 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j"] Oct 30 01:21:35.172174 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:35.172141 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j"] Oct 30 01:21:35.196405 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:35.196355 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-29145c/sub-test-model-29145c-kserve-8444cf8dbf-8xq4j" podUID="31ba3725-cc95-463d-9580-513e3a0354c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.131:8000/ready\": context deadline exceeded" Oct 30 01:21:35.826509 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:35.826470 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31ba3725-cc95-463d-9580-513e3a0354c0" path="/var/lib/kubelet/pods/31ba3725-cc95-463d-9580-513e3a0354c0/volumes" Oct 30 01:21:36.049425 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:36.049388 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf"] Oct 30 01:21:36.050211 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:36.050189 5300 cpu_manager.go:414] "containerMap: removing container" podUID="b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7" containerName="main" Oct 30 01:21:36.050211 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:36.050210 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7" containerName="main" Oct 30 01:21:36.050400 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:36.050287 5300 cpu_manager.go:414] "containerMap: removing container" podUID="31ba3725-cc95-463d-9580-513e3a0354c0" containerName="main" Oct 30 01:21:36.050400 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:36.050296 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ba3725-cc95-463d-9580-513e3a0354c0" containerName="main" Oct 30 01:21:36.050516 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:36.050482 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2388217-f8d3-4bd7-9f6a-dc0bfd7db1d7" containerName="main" Oct 30 01:21:36.050516 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:36.050506 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="31ba3725-cc95-463d-9580-513e3a0354c0" containerName="main" Oct 30 01:21:36.055825 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:36.055804 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" Oct 30 01:21:36.075609 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:36.075577 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf"] Oct 30 01:21:36.085833 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:36.081903 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-999ff9-serving-cert\"" Oct 30 01:21:36.130575 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:36.130415 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:36.174093 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:36.174051 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49bcda0b-9448-4e57-8328-220ee6208164-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf\" (UID: \"49bcda0b-9448-4e57-8328-220ee6208164\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" Oct 30 01:21:36.174371 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:36.174349 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcvck\" (UniqueName: \"kubernetes.io/projected/49bcda0b-9448-4e57-8328-220ee6208164-kube-api-access-hcvck\") pod \"maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf\" (UID: \"49bcda0b-9448-4e57-8328-220ee6208164\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" Oct 30 01:21:36.174497 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:36.174483 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49bcda0b-9448-4e57-8328-220ee6208164-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf\" (UID: \"49bcda0b-9448-4e57-8328-220ee6208164\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" Oct 30 01:21:36.174772 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:36.174738 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49bcda0b-9448-4e57-8328-220ee6208164-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf\" (UID: \"49bcda0b-9448-4e57-8328-220ee6208164\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" Oct 30 01:21:36.236692 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:36.236657 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:36.279735 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:36.279671 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49bcda0b-9448-4e57-8328-220ee6208164-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf\" (UID: \"49bcda0b-9448-4e57-8328-220ee6208164\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" Oct 30 01:21:36.280330 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:36.280300 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49bcda0b-9448-4e57-8328-220ee6208164-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf\" (UID: \"49bcda0b-9448-4e57-8328-220ee6208164\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" Oct 30 01:21:36.280523 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:36.280502 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcvck\" (UniqueName: \"kubernetes.io/projected/49bcda0b-9448-4e57-8328-220ee6208164-kube-api-access-hcvck\") pod \"maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf\" (UID: \"49bcda0b-9448-4e57-8328-220ee6208164\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" Oct 30 01:21:36.280677 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:36.280660 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49bcda0b-9448-4e57-8328-220ee6208164-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf\" (UID: \"49bcda0b-9448-4e57-8328-220ee6208164\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" Oct 30 01:21:36.280940 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:21:36.280695 5300 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-serving-cert: secret "maas-api-e2e-shared-b-w4-999ff9-serving-cert" not found Oct 30 01:21:36.281125 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:21:36.281109 5300 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49bcda0b-9448-4e57-8328-220ee6208164-maas-api-tls podName:49bcda0b-9448-4e57-8328-220ee6208164 nodeName:}" failed. No retries permitted until 2026-10-30 01:21:36.781084834 +0000 UTC m=+2297.096953161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/49bcda0b-9448-4e57-8328-220ee6208164-maas-api-tls") pod "maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" (UID: "49bcda0b-9448-4e57-8328-220ee6208164") : secret "maas-api-e2e-shared-b-w4-999ff9-serving-cert" not found Oct 30 01:21:36.282335 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:36.281879 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49bcda0b-9448-4e57-8328-220ee6208164-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf\" (UID: \"49bcda0b-9448-4e57-8328-220ee6208164\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" Oct 30 01:21:36.287841 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:36.283590 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49bcda0b-9448-4e57-8328-220ee6208164-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf\" (UID: \"49bcda0b-9448-4e57-8328-220ee6208164\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" Oct 30 01:21:36.328187 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:36.326283 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcvck\" (UniqueName: \"kubernetes.io/projected/49bcda0b-9448-4e57-8328-220ee6208164-kube-api-access-hcvck\") pod \"maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf\" (UID: \"49bcda0b-9448-4e57-8328-220ee6208164\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" Oct 30 01:21:36.795074 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:36.795035 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49bcda0b-9448-4e57-8328-220ee6208164-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf\" (UID: \"49bcda0b-9448-4e57-8328-220ee6208164\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" Oct 30 01:21:36.795272 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:21:36.795209 5300 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-serving-cert: secret "maas-api-e2e-shared-b-w4-999ff9-serving-cert" not found Oct 30 01:21:36.795319 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:21:36.795298 5300 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49bcda0b-9448-4e57-8328-220ee6208164-maas-api-tls podName:49bcda0b-9448-4e57-8328-220ee6208164 nodeName:}" failed. No retries permitted until 2026-10-30 01:21:37.795279369 +0000 UTC m=+2298.111147677 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/49bcda0b-9448-4e57-8328-220ee6208164-maas-api-tls") pod "maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" (UID: "49bcda0b-9448-4e57-8328-220ee6208164") : secret "maas-api-e2e-shared-b-w4-999ff9-serving-cert" not found Oct 30 01:21:36.908345 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:36.908293 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:37.826894 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:37.826859 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49bcda0b-9448-4e57-8328-220ee6208164-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf\" (UID: \"49bcda0b-9448-4e57-8328-220ee6208164\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" Oct 30 01:21:37.829364 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:37.829337 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49bcda0b-9448-4e57-8328-220ee6208164-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf\" (UID: \"49bcda0b-9448-4e57-8328-220ee6208164\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" Oct 30 01:21:37.871863 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:37.871808 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" Oct 30 01:21:38.043228 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:38.043147 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:38.188835 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:38.188580 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf"] Oct 30 01:21:39.040513 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:39.040482 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr" Oct 30 01:21:39.201326 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:39.201287 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" event={"ID":"49bcda0b-9448-4e57-8328-220ee6208164","Type":"ContainerStarted","Data":"decfd3cf0ee4eb4679b1d625b20e5c8f602c18503495394647387a4b07830fc1"} Oct 30 01:21:39.201537 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:39.201351 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" event={"ID":"49bcda0b-9448-4e57-8328-220ee6208164","Type":"ContainerStarted","Data":"4097b44f7a4d37a0d7f27d11918289bdc93146a8706e697806b153da1080aab5"} Oct 30 01:21:39.201537 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:39.201379 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" Oct 30 01:21:39.246130 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:39.246082 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" podStartSLOduration=2.931730585 podStartE2EDuration="3.246068128s" totalImagesPullingTime="314.337543ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:39.243954113 +0000 UTC m=+2299.559822439" watchObservedRunningTime="2026-10-30 01:21:39.246068128 +0000 UTC m=+2299.561936490" Oct 30 01:21:39.561332 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:39.561274 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:41.117013 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:41.116976 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:41.343394 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:41.343363 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:42.553346 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:42.553309 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:44.050930 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:44.050892 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:45.213015 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.212984 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx"] Oct 30 01:21:45.217944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.217920 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" Oct 30 01:21:45.218358 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.218332 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" Oct 30 01:21:45.238398 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.238368 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx"] Oct 30 01:21:45.245470 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.245439 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-886adb\"/\"openshift-service-ca.crt\"" Oct 30 01:21:45.245667 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.245649 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-886adb\"/\"kube-root-ca.crt\"" Oct 30 01:21:45.245889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.245844 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-886adb\"/\"default-dockercfg-plkzf\"" Oct 30 01:21:45.249907 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.249878 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:45.250603 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.250580 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-886adb\"/\"sub-test-model-886adb-kserve-self-signed-certs\"" Oct 30 01:21:45.347635 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.347603 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-model-cache\") pod \"sub-test-model-886adb-kserve-844f56697f-zlgnx\" (UID: \"3f4fe743-0164-4a05-a608-5d7078948f3a\") " pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" Oct 30 01:21:45.347836 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.347653 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-dshm\") pod \"sub-test-model-886adb-kserve-844f56697f-zlgnx\" (UID: \"3f4fe743-0164-4a05-a608-5d7078948f3a\") " pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" Oct 30 01:21:45.347836 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.347687 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f4fe743-0164-4a05-a608-5d7078948f3a-tls-certs\") pod \"sub-test-model-886adb-kserve-844f56697f-zlgnx\" (UID: \"3f4fe743-0164-4a05-a608-5d7078948f3a\") " pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" Oct 30 01:21:45.348041 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.348018 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4pbz\" (UniqueName: \"kubernetes.io/projected/3f4fe743-0164-4a05-a608-5d7078948f3a-kube-api-access-m4pbz\") pod \"sub-test-model-886adb-kserve-844f56697f-zlgnx\" (UID: \"3f4fe743-0164-4a05-a608-5d7078948f3a\") " pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" Oct 30 01:21:45.348153 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.348130 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-home\") pod \"sub-test-model-886adb-kserve-844f56697f-zlgnx\" (UID: \"3f4fe743-0164-4a05-a608-5d7078948f3a\") " pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" Oct 30 01:21:45.348336 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.348315 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-tmp-dir\") pod \"sub-test-model-886adb-kserve-844f56697f-zlgnx\" (UID: \"3f4fe743-0164-4a05-a608-5d7078948f3a\") " pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" Oct 30 01:21:45.450459 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.450416 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-model-cache\") pod \"sub-test-model-886adb-kserve-844f56697f-zlgnx\" (UID: \"3f4fe743-0164-4a05-a608-5d7078948f3a\") " pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" Oct 30 01:21:45.451258 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.451228 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-dshm\") pod \"sub-test-model-886adb-kserve-844f56697f-zlgnx\" (UID: \"3f4fe743-0164-4a05-a608-5d7078948f3a\") " pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" Oct 30 01:21:45.451419 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.451288 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f4fe743-0164-4a05-a608-5d7078948f3a-tls-certs\") pod \"sub-test-model-886adb-kserve-844f56697f-zlgnx\" (UID: \"3f4fe743-0164-4a05-a608-5d7078948f3a\") " pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" Oct 30 01:21:45.451419 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.451141 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-model-cache\") pod \"sub-test-model-886adb-kserve-844f56697f-zlgnx\" (UID: \"3f4fe743-0164-4a05-a608-5d7078948f3a\") " pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" Oct 30 01:21:45.451538 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.451488 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4pbz\" (UniqueName: \"kubernetes.io/projected/3f4fe743-0164-4a05-a608-5d7078948f3a-kube-api-access-m4pbz\") pod \"sub-test-model-886adb-kserve-844f56697f-zlgnx\" (UID: \"3f4fe743-0164-4a05-a608-5d7078948f3a\") " pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" Oct 30 01:21:45.451538 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.451532 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-home\") pod \"sub-test-model-886adb-kserve-844f56697f-zlgnx\" (UID: \"3f4fe743-0164-4a05-a608-5d7078948f3a\") " pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" Oct 30 01:21:45.451636 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.451579 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-tmp-dir\") pod \"sub-test-model-886adb-kserve-844f56697f-zlgnx\" (UID: \"3f4fe743-0164-4a05-a608-5d7078948f3a\") " pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" Oct 30 01:21:45.451978 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.451949 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-home\") pod \"sub-test-model-886adb-kserve-844f56697f-zlgnx\" (UID: \"3f4fe743-0164-4a05-a608-5d7078948f3a\") " pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" Oct 30 01:21:45.452094 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.452061 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-tmp-dir\") pod \"sub-test-model-886adb-kserve-844f56697f-zlgnx\" (UID: \"3f4fe743-0164-4a05-a608-5d7078948f3a\") " pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" Oct 30 01:21:45.464862 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.454493 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-dshm\") pod \"sub-test-model-886adb-kserve-844f56697f-zlgnx\" (UID: \"3f4fe743-0164-4a05-a608-5d7078948f3a\") " pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" Oct 30 01:21:45.464862 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.454774 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f4fe743-0164-4a05-a608-5d7078948f3a-tls-certs\") pod \"sub-test-model-886adb-kserve-844f56697f-zlgnx\" (UID: \"3f4fe743-0164-4a05-a608-5d7078948f3a\") " pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" Oct 30 01:21:45.480553 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.480486 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4pbz\" (UniqueName: \"kubernetes.io/projected/3f4fe743-0164-4a05-a608-5d7078948f3a-kube-api-access-m4pbz\") pod \"sub-test-model-886adb-kserve-844f56697f-zlgnx\" (UID: \"3f4fe743-0164-4a05-a608-5d7078948f3a\") " pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" Oct 30 01:21:45.538694 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.538662 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" Oct 30 01:21:45.546980 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.546952 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:45.846929 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:45.846900 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx"] Oct 30 01:21:45.848241 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:21:45.848209 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f4fe743_0164_4a05_a608_5d7078948f3a.slice/crio-34f8b4355f576e6fa6a8739a31d9b329fbc53891d1510257c4fcb3dc7545a080 WatchSource:0}: Error finding container 34f8b4355f576e6fa6a8739a31d9b329fbc53891d1510257c4fcb3dc7545a080: Status 404 returned error can't find the container with id 34f8b4355f576e6fa6a8739a31d9b329fbc53891d1510257c4fcb3dc7545a080 Oct 30 01:21:46.260029 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:46.259995 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" event={"ID":"3f4fe743-0164-4a05-a608-5d7078948f3a","Type":"ContainerStarted","Data":"b2a556e509bc043c32e850c50e09d3f69f4a794b6d74d593738e44e842463c12"} Oct 30 01:21:46.260409 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:46.260039 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" event={"ID":"3f4fe743-0164-4a05-a608-5d7078948f3a","Type":"ContainerStarted","Data":"34f8b4355f576e6fa6a8739a31d9b329fbc53891d1510257c4fcb3dc7545a080"} Oct 30 01:21:46.303738 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:46.303668 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" podStartSLOduration=1.303649217 podStartE2EDuration="1.303649217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:46.30120861 +0000 UTC m=+2306.617076938" watchObservedRunningTime="2026-10-30 01:21:46.303649217 +0000 UTC m=+2306.619517543" Oct 30 01:21:46.539526 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:46.539439 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" Oct 30 01:21:47.213973 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.213933 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx"] Oct 30 01:21:47.218050 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.218032 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.244152 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.243328 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx"] Oct 30 01:21:47.255019 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.254983 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:47.280848 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.280809 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.281225 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.280951 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24445fd3-1ef0-4523-9548-7d1400bcb32b-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.281225 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.281032 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.281225 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.281081 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg5hh\" (UniqueName: \"kubernetes.io/projected/24445fd3-1ef0-4523-9548-7d1400bcb32b-kube-api-access-dg5hh\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.281225 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.281152 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.281358 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.281241 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.281358 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.281274 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.281358 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.281325 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.281451 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.281360 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.383640 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.383603 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24445fd3-1ef0-4523-9548-7d1400bcb32b-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.383901 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.383694 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.383901 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.383738 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg5hh\" (UniqueName: \"kubernetes.io/projected/24445fd3-1ef0-4523-9548-7d1400bcb32b-kube-api-access-dg5hh\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.383901 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.383820 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.383901 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.383901 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.384111 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.383927 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.384111 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.383967 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.384111 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.384000 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.384111 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.384049 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.385053 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.385020 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.385176 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.385140 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.385387 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.385358 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24445fd3-1ef0-4523-9548-7d1400bcb32b-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.385654 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.385632 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.385717 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.385643 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.388111 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.388082 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.388283 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.388259 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.405395 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.405366 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.405897 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.405875 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg5hh\" (UniqueName: \"kubernetes.io/projected/24445fd3-1ef0-4523-9548-7d1400bcb32b-kube-api-access-dg5hh\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.535359 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.535320 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:47.862145 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.862111 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx"] Oct 30 01:21:47.862252 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:21:47.862115 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24445fd3_1ef0_4523_9548_7d1400bcb32b.slice/crio-b13f880b6478fa435561223cdf0597ec54701a1c7d5d21eaa3b63e97db1be3bb WatchSource:0}: Error finding container b13f880b6478fa435561223cdf0597ec54701a1c7d5d21eaa3b63e97db1be3bb: Status 404 returned error can't find the container with id b13f880b6478fa435561223cdf0597ec54701a1c7d5d21eaa3b63e97db1be3bb Oct 30 01:21:47.865395 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.865362 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:21:47.865460 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.865434 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:21:47.865504 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:47.865464 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:21:48.281952 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:48.281911 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" event={"ID":"24445fd3-1ef0-4523-9548-7d1400bcb32b","Type":"ContainerStarted","Data":"1c809ade95391c4ccf74e39c84d8c0d7db863564a3e391a193368d41af1da592"} Oct 30 01:21:48.282438 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:48.281965 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" event={"ID":"24445fd3-1ef0-4523-9548-7d1400bcb32b","Type":"ContainerStarted","Data":"b13f880b6478fa435561223cdf0597ec54701a1c7d5d21eaa3b63e97db1be3bb"} Oct 30 01:21:48.326260 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:48.326202 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" podStartSLOduration=1.326188925 podStartE2EDuration="1.326188925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:48.3248103 +0000 UTC m=+2308.640678627" watchObservedRunningTime="2026-10-30 01:21:48.326188925 +0000 UTC m=+2308.642057250" Oct 30 01:21:48.536461 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:48.536376 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:48.538117 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:48.538087 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 01:21:48.538227 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:48.538147 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" podUID="24445fd3-1ef0-4523-9548-7d1400bcb32b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 01:21:48.684172 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:48.684139 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:49.543247 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:49.543202 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 01:21:49.543652 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:49.543275 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" podUID="24445fd3-1ef0-4523-9548-7d1400bcb32b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 01:21:50.194258 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:50.194223 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:50.542835 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:50.542213 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 01:21:50.542835 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:50.542281 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" podUID="24445fd3-1ef0-4523-9548-7d1400bcb32b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 01:21:51.536232 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:51.536197 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 01:21:51.543960 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:51.543879 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" podUID="24445fd3-1ef0-4523-9548-7d1400bcb32b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 01:21:51.595291 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:21:51.595249 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:21:52.067030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:52.066990 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:52.539787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:52.538621 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 01:21:52.539787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:52.538687 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" podUID="24445fd3-1ef0-4523-9548-7d1400bcb32b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 01:21:53.547787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.536168 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 01:21:53.547787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.536235 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" podUID="24445fd3-1ef0-4523-9548-7d1400bcb32b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 01:21:53.581553 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.581519 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw"] Oct 30 01:21:53.586752 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.586717 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" Oct 30 01:21:53.598296 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.597278 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx"] Oct 30 01:21:53.603118 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.603100 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:53.605978 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.605953 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v"] Oct 30 01:21:53.607078 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.607055 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-e92648-serving-cert\"" Oct 30 01:21:53.611661 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.611641 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.659787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.624907 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw"] Oct 30 01:21:53.764020 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.763983 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v"] Oct 30 01:21:53.764217 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.758353 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:53.764217 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.712639 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.764217 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.764157 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw\" (UID: \"016eaf4d-095e-4eec-9cae-2c3cb12e13b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" Oct 30 01:21:53.764388 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.764229 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:53.764388 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.764264 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhjkb\" (UniqueName: \"kubernetes.io/projected/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-kube-api-access-bhjkb\") pod \"maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw\" (UID: \"016eaf4d-095e-4eec-9cae-2c3cb12e13b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" Oct 30 01:21:53.764388 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.764315 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:53.764388 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.764346 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw\" (UID: \"016eaf4d-095e-4eec-9cae-2c3cb12e13b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" Oct 30 01:21:53.764609 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.764402 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p8ml\" (UniqueName: \"kubernetes.io/projected/b03e0096-ff6f-4e17-b213-72a40602bf18-kube-api-access-4p8ml\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:53.764609 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.764426 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw\" (UID: \"016eaf4d-095e-4eec-9cae-2c3cb12e13b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" Oct 30 01:21:53.764609 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.764454 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.764609 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.764479 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:53.764609 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.764512 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e79fad-5a98-4840-9218-41ddff154453-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.764609 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.764601 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x75pz\" (UniqueName: \"kubernetes.io/projected/86e79fad-5a98-4840-9218-41ddff154453-kube-api-access-x75pz\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.765331 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.764629 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:53.765331 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.764676 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:53.765331 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.764707 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.765331 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.764799 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.765331 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.764826 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:53.765331 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.765008 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:53.765331 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.765146 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e79fad-5a98-4840-9218-41ddff154453-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.765331 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.765204 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b03e0096-ff6f-4e17-b213-72a40602bf18-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:53.765331 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.765236 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86e79fad-5a98-4840-9218-41ddff154453-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.765331 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.765316 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.775663 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.775635 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx"] Oct 30 01:21:53.867029 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.866940 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:53.867029 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.867024 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e79fad-5a98-4840-9218-41ddff154453-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.867289 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.867055 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b03e0096-ff6f-4e17-b213-72a40602bf18-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:53.867289 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.867074 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86e79fad-5a98-4840-9218-41ddff154453-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.867289 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.867110 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.867289 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.867154 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.867289 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.867170 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw\" (UID: \"016eaf4d-095e-4eec-9cae-2c3cb12e13b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" Oct 30 01:21:53.867289 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.867196 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:53.867289 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.867223 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhjkb\" (UniqueName: \"kubernetes.io/projected/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-kube-api-access-bhjkb\") pod \"maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw\" (UID: \"016eaf4d-095e-4eec-9cae-2c3cb12e13b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" Oct 30 01:21:53.867289 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.867263 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:53.867682 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.867291 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw\" (UID: \"016eaf4d-095e-4eec-9cae-2c3cb12e13b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" Oct 30 01:21:53.867682 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.867339 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p8ml\" (UniqueName: \"kubernetes.io/projected/b03e0096-ff6f-4e17-b213-72a40602bf18-kube-api-access-4p8ml\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:53.867682 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.867362 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw\" (UID: \"016eaf4d-095e-4eec-9cae-2c3cb12e13b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" Oct 30 01:21:53.867682 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.867388 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.867682 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.867412 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:53.867682 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.867445 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e79fad-5a98-4840-9218-41ddff154453-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.867682 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.867524 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x75pz\" (UniqueName: \"kubernetes.io/projected/86e79fad-5a98-4840-9218-41ddff154453-kube-api-access-x75pz\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.867682 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.867553 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:53.867682 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.867598 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:53.867682 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.867628 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.867682 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.867678 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.868280 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.867702 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:53.868280 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.867797 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.868280 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.867882 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.868280 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.867969 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b03e0096-ff6f-4e17-b213-72a40602bf18-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:53.868280 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:21:53.868020 5300 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-serving-cert: secret "maas-api-e2e-shared-b-w6-e92648-serving-cert" not found Oct 30 01:21:53.868280 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.868042 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw\" (UID: \"016eaf4d-095e-4eec-9cae-2c3cb12e13b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" Oct 30 01:21:53.868280 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:21:53.868077 5300 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-maas-api-tls podName:016eaf4d-095e-4eec-9cae-2c3cb12e13b6 nodeName:}" failed. No retries permitted until 2026-10-30 01:21:54.368059688 +0000 UTC m=+2314.683928017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-maas-api-tls") pod "maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" (UID: "016eaf4d-095e-4eec-9cae-2c3cb12e13b6") : secret "maas-api-e2e-shared-b-w6-e92648-serving-cert" not found Oct 30 01:21:53.868703 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.868336 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:53.868703 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.868365 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:53.868703 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.868530 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-data\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:53.868703 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.868629 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-workload-certs\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.868703 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.868661 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-credential-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.868997 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.868810 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e79fad-5a98-4840-9218-41ddff154453-istiod-ca-cert\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.868997 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.868827 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-workload-socket\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:53.870488 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.870453 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e79fad-5a98-4840-9218-41ddff154453-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.870655 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.870592 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw\" (UID: \"016eaf4d-095e-4eec-9cae-2c3cb12e13b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" Oct 30 01:21:53.870655 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.870645 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:53.871092 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.871072 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-istio-envoy\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.872052 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.872029 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-podinfo\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:53.878029 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.878005 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:53.889803 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.889742 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhjkb\" (UniqueName: \"kubernetes.io/projected/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-kube-api-access-bhjkb\") pod \"maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw\" (UID: \"016eaf4d-095e-4eec-9cae-2c3cb12e13b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" Oct 30 01:21:53.890161 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.890137 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86e79fad-5a98-4840-9218-41ddff154453-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.890250 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.890199 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-token\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:53.890838 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.890817 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x75pz\" (UniqueName: \"kubernetes.io/projected/86e79fad-5a98-4840-9218-41ddff154453-kube-api-access-x75pz\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:53.891339 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:53.891322 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p8ml\" (UniqueName: \"kubernetes.io/projected/b03e0096-ff6f-4e17-b213-72a40602bf18-kube-api-access-4p8ml\") pod \"e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:54.068472 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:54.068423 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:54.082264 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:54.082074 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:54.377734 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:54.377693 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw\" (UID: \"016eaf4d-095e-4eec-9cae-2c3cb12e13b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" Oct 30 01:21:54.377950 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:21:54.377910 5300 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-serving-cert: secret "maas-api-e2e-shared-b-w6-e92648-serving-cert" not found Oct 30 01:21:54.378031 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:21:54.377989 5300 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-maas-api-tls podName:016eaf4d-095e-4eec-9cae-2c3cb12e13b6 nodeName:}" failed. No retries permitted until 2026-10-30 01:21:55.377967758 +0000 UTC m=+2315.693836076 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-maas-api-tls") pod "maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" (UID: "016eaf4d-095e-4eec-9cae-2c3cb12e13b6") : secret "maas-api-e2e-shared-b-w6-e92648-serving-cert" not found Oct 30 01:21:54.441857 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:54.441615 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx"] Oct 30 01:21:54.449805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:54.449721 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:21:54.449932 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:54.449911 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:21:54.450019 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:54.450000 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:21:54.456057 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:54.456020 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v"] Oct 30 01:21:54.468657 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:54.468624 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:21:54.468752 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:54.468691 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:21:54.468752 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:54.468720 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:21:54.535753 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:54.535718 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 01:21:54.535840 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:54.535777 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" podUID="24445fd3-1ef0-4523-9548-7d1400bcb32b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 01:21:55.367370 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:55.367328 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" event={"ID":"86e79fad-5a98-4840-9218-41ddff154453","Type":"ContainerStarted","Data":"f34bf18efdb30601371cf1f26c192b11529768258198d42218262f19ef534a91"} Oct 30 01:21:55.367909 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:55.367398 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" event={"ID":"86e79fad-5a98-4840-9218-41ddff154453","Type":"ContainerStarted","Data":"1ae9962dcb294c01e6154b96b6356cd17d96c294e6b4b43c44830627ff05366c"} Oct 30 01:21:55.369220 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:55.369193 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" event={"ID":"b03e0096-ff6f-4e17-b213-72a40602bf18","Type":"ContainerStarted","Data":"3c3ad4a44e7fedf7e8ba15ed1fae3b7be55b57b295a747527c69ea56d5a017e6"} Oct 30 01:21:55.369338 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:55.369244 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" event={"ID":"b03e0096-ff6f-4e17-b213-72a40602bf18","Type":"ContainerStarted","Data":"1dacdfd4ae81f26cdbc4af3a5c4422e7e74629dda69b57da4ec921d895bea535"} Oct 30 01:21:55.396922 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:55.396892 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw\" (UID: \"016eaf4d-095e-4eec-9cae-2c3cb12e13b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" Oct 30 01:21:55.397071 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:21:55.397026 5300 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-serving-cert: secret "maas-api-e2e-shared-b-w6-e92648-serving-cert" not found Oct 30 01:21:55.397121 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:21:55.397087 5300 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-maas-api-tls podName:016eaf4d-095e-4eec-9cae-2c3cb12e13b6 nodeName:}" failed. No retries permitted until 2026-10-30 01:21:57.397073194 +0000 UTC m=+2317.712941497 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-maas-api-tls") pod "maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" (UID: "016eaf4d-095e-4eec-9cae-2c3cb12e13b6") : secret "maas-api-e2e-shared-b-w6-e92648-serving-cert" not found Oct 30 01:21:55.429717 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:55.429668 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" podStartSLOduration=2.429655491 podStartE2EDuration="2.429655491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:55.428131426 +0000 UTC m=+2315.743999773" watchObservedRunningTime="2026-10-30 01:21:55.429655491 +0000 UTC m=+2315.745523816" Oct 30 01:21:55.481848 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:55.480053 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" podStartSLOduration=2.480032709 podStartE2EDuration="2.480032709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:55.475003182 +0000 UTC m=+2315.790871531" watchObservedRunningTime="2026-10-30 01:21:55.480032709 +0000 UTC m=+2315.795901035" Oct 30 01:21:55.486779 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:55.483719 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:55.546786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:55.544615 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" Oct 30 01:21:55.555321 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:55.555282 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:55.556934 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:55.556908 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:55.557911 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:55.557742 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:21:55.567486 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:55.567468 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" Oct 30 01:21:55.573950 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:55.573927 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:56.069373 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:56.069337 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:21:56.072028 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:56.071998 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 01:21:56.072195 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:56.072052 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" podUID="b03e0096-ff6f-4e17-b213-72a40602bf18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 01:21:56.083229 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:56.083202 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:21:56.085125 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:56.085099 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 01:21:56.085256 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:56.085145 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" podUID="86e79fad-5a98-4840-9218-41ddff154453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 01:21:56.409025 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:56.408944 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" Oct 30 01:21:57.071790 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:57.069106 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 01:21:57.071790 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:57.069171 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" podUID="b03e0096-ff6f-4e17-b213-72a40602bf18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 01:21:57.083170 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:57.083143 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 01:21:57.083403 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:57.083381 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" podUID="86e79fad-5a98-4840-9218-41ddff154453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 01:21:57.302658 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:57.302624 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:57.470533 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:57.470488 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw\" (UID: \"016eaf4d-095e-4eec-9cae-2c3cb12e13b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" Oct 30 01:21:57.474669 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:57.474628 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw\" (UID: \"016eaf4d-095e-4eec-9cae-2c3cb12e13b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" Oct 30 01:21:57.505935 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:57.505896 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" Oct 30 01:21:57.973903 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:57.973868 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw"] Oct 30 01:21:57.975491 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:21:57.975469 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod016eaf4d_095e_4eec_9cae_2c3cb12e13b6.slice/crio-0408076955c857e5da697777a13cd87e294a11f680e7b9b856bbfd5f253ea7c6 WatchSource:0}: Error finding container 0408076955c857e5da697777a13cd87e294a11f680e7b9b856bbfd5f253ea7c6: Status 404 returned error can't find the container with id 0408076955c857e5da697777a13cd87e294a11f680e7b9b856bbfd5f253ea7c6 Oct 30 01:21:58.069209 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:58.069164 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 01:21:58.069434 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:58.069238 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" podUID="b03e0096-ff6f-4e17-b213-72a40602bf18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 01:21:58.083659 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:58.083626 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 01:21:58.083849 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:58.083682 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" podUID="86e79fad-5a98-4840-9218-41ddff154453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 01:21:58.414108 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:58.414071 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" event={"ID":"016eaf4d-095e-4eec-9cae-2c3cb12e13b6","Type":"ContainerStarted","Data":"0408076955c857e5da697777a13cd87e294a11f680e7b9b856bbfd5f253ea7c6"} Oct 30 01:21:58.775371 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:58.775333 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:59.072837 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:59.072170 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 01:21:59.072837 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:59.072237 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" podUID="b03e0096-ff6f-4e17-b213-72a40602bf18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 01:21:59.093832 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:59.089257 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 01:21:59.093832 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:59.089331 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" podUID="86e79fad-5a98-4840-9218-41ddff154453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 01:21:59.293595 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:59.293562 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:21:59.482340 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:59.482299 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" event={"ID":"016eaf4d-095e-4eec-9cae-2c3cb12e13b6","Type":"ContainerStarted","Data":"2c6ac8c3ed1c9398988a5417c7bbe9a8a323bde6bd3d14f2faed04fa5c7cfd76"} Oct 30 01:21:59.482533 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:59.482392 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" Oct 30 01:21:59.572676 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:59.572599 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" podStartSLOduration=6.159571628 podStartE2EDuration="6.572577886s" totalImagesPullingTime="413.006258ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:59.540107887 +0000 UTC m=+2319.855976210" watchObservedRunningTime="2026-10-30 01:21:59.572577886 +0000 UTC m=+2319.888446214" Oct 30 01:21:59.680907 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:21:59.680732 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:00.070786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:00.069135 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 01:22:00.070786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:00.069182 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" podUID="b03e0096-ff6f-4e17-b213-72a40602bf18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 01:22:00.083136 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:00.083056 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 01:22:00.083430 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:00.083388 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" podUID="86e79fad-5a98-4840-9218-41ddff154453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 01:22:00.175409 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:00.175374 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:00.839786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:00.831208 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:01.045313 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:01.045280 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:01.076821 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:01.076056 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 01:22:01.076821 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:01.076160 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" podUID="b03e0096-ff6f-4e17-b213-72a40602bf18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 01:22:01.084085 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:01.083930 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 01:22:01.084085 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:01.083998 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" podUID="86e79fad-5a98-4840-9218-41ddff154453" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 01:22:01.155365 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:01.155272 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:02.089477 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.089441 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:22:02.090150 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.090067 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:22:02.091052 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.091022 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:22:02.110903 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.110875 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:22:02.112809 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.111287 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:22:02.112809 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.112345 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:22:02.244415 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.244379 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq"] Oct 30 01:22:02.249510 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.249480 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.271978 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.271949 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq"] Oct 30 01:22:02.358852 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.358715 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.359001 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.358945 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.359143 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.359120 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xskkz\" (UniqueName: \"kubernetes.io/projected/cfadd1c5-3001-4ebb-8156-46628aa2e722-kube-api-access-xskkz\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.359341 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.359315 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.359467 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.359366 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.359513 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.359483 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.359614 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.359594 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfadd1c5-3001-4ebb-8156-46628aa2e722-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.359666 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.359651 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.359703 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.359683 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.461228 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.461191 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.461228 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.461235 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.461448 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.461277 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.461448 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.461358 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfadd1c5-3001-4ebb-8156-46628aa2e722-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.461448 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.461394 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.461448 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.461416 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.461651 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.461520 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.461651 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.461640 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-data\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.461800 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.461668 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.461800 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.461711 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-workload-certs\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.461800 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.461730 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xskkz\" (UniqueName: \"kubernetes.io/projected/cfadd1c5-3001-4ebb-8156-46628aa2e722-kube-api-access-xskkz\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.462047 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.462025 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-credential-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.462145 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.462122 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-workload-socket\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.462302 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.462282 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfadd1c5-3001-4ebb-8156-46628aa2e722-istiod-ca-cert\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.463812 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.463791 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-envoy\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.463951 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.463933 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-podinfo\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.482234 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.482205 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-token\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.482365 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.482279 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xskkz\" (UniqueName: \"kubernetes.io/projected/cfadd1c5-3001-4ebb-8156-46628aa2e722-kube-api-access-xskkz\") pod \"e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.574430 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.574395 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:02.676858 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.676441 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:02.761617 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.761580 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:02.861770 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.861728 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:02.901499 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.901448 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq"] Oct 30 01:22:02.901795 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:22:02.901704 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfadd1c5_3001_4ebb_8156_46628aa2e722.slice/crio-d6444f7a76adeadf0c0d60c826bb5d066086a791be7cf3274b328e264dce3e8d WatchSource:0}: Error finding container d6444f7a76adeadf0c0d60c826bb5d066086a791be7cf3274b328e264dce3e8d: Status 404 returned error can't find the container with id d6444f7a76adeadf0c0d60c826bb5d066086a791be7cf3274b328e264dce3e8d Oct 30 01:22:02.905233 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.905183 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:22:02.905333 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.905271 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:22:02.905333 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:02.905312 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:22:03.561490 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:03.561451 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" event={"ID":"cfadd1c5-3001-4ebb-8156-46628aa2e722","Type":"ContainerStarted","Data":"49d0fc62354fd7ecf47bf54b59cf530f3b55936f6b95a5035ffa577fc90c4921"} Oct 30 01:22:03.561943 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:03.561517 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" event={"ID":"cfadd1c5-3001-4ebb-8156-46628aa2e722","Type":"ContainerStarted","Data":"d6444f7a76adeadf0c0d60c826bb5d066086a791be7cf3274b328e264dce3e8d"} Oct 30 01:22:03.575138 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:03.575110 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:03.576955 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:03.576930 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 01:22:03.577092 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:03.576976 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" podUID="cfadd1c5-3001-4ebb-8156-46628aa2e722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 01:22:03.608381 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:03.608332 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" podStartSLOduration=1.608317051 podStartE2EDuration="1.608317051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:03.605193387 +0000 UTC m=+2323.921061712" watchObservedRunningTime="2026-10-30 01:22:03.608317051 +0000 UTC m=+2323.924185379" Oct 30 01:22:04.563829 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:04.563780 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:04.575186 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:04.575149 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 01:22:04.575402 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:04.575224 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" podUID="cfadd1c5-3001-4ebb-8156-46628aa2e722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 01:22:05.086483 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:05.086343 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:05.187107 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:05.187074 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:05.509211 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:05.509185 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" Oct 30 01:22:05.575282 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:05.575253 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 01:22:05.575874 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:05.575311 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" podUID="cfadd1c5-3001-4ebb-8156-46628aa2e722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 01:22:06.404954 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:06.404920 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:06.575036 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:06.575003 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 01:22:06.575243 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:06.575056 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" podUID="cfadd1c5-3001-4ebb-8156-46628aa2e722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 01:22:06.877875 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:06.877835 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:06.973611 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:06.973569 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:07.576830 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:07.575118 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 01:22:07.576830 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:07.575180 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" podUID="cfadd1c5-3001-4ebb-8156-46628aa2e722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 01:22:08.171595 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:08.171528 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:08.575101 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:08.575065 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 01:22:08.575268 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:08.575123 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" podUID="cfadd1c5-3001-4ebb-8156-46628aa2e722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 01:22:09.579953 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:09.579915 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:09.580380 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:09.580290 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:09.580977 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:09.580956 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:22:09.634429 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:09.634398 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:11.341261 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:11.341225 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:13.001801 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:13.001745 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:14.454141 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:14.454107 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:15.959565 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:15.959527 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:17.631026 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:17.630992 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:19.343512 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:19.343482 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:20.912591 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:20.912541 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:21.595602 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:22:21.595561 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:22:21.757803 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:21.757771 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:22.579994 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:22.579956 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:24.237919 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:24.237855 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps"] Oct 30 01:22:24.403242 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:24.403205 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:24.483229 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:24.483184 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps"] Oct 30 01:22:24.483428 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:24.483307 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" Oct 30 01:22:24.513815 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:24.513773 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hljx2\" (UniqueName: \"kubernetes.io/projected/0957caa6-ff6b-4cf8-be88-717b6434a22f-kube-api-access-hljx2\") pod \"auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps\" (UID: \"0957caa6-ff6b-4cf8-be88-717b6434a22f\") " pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" Oct 30 01:22:24.514088 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:24.514062 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-tmp-dir\") pod \"auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps\" (UID: \"0957caa6-ff6b-4cf8-be88-717b6434a22f\") " pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" Oct 30 01:22:24.514217 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:24.514201 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-dshm\") pod \"auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps\" (UID: \"0957caa6-ff6b-4cf8-be88-717b6434a22f\") " pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" Oct 30 01:22:24.514331 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:24.514318 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-model-cache\") pod \"auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps\" (UID: \"0957caa6-ff6b-4cf8-be88-717b6434a22f\") " pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" Oct 30 01:22:24.514394 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:24.514349 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0957caa6-ff6b-4cf8-be88-717b6434a22f-tls-certs\") pod \"auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps\" (UID: \"0957caa6-ff6b-4cf8-be88-717b6434a22f\") " pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" Oct 30 01:22:24.514552 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:24.514527 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-home\") pod \"auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps\" (UID: \"0957caa6-ff6b-4cf8-be88-717b6434a22f\") " pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" Oct 30 01:22:24.515729 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:24.515706 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-e2d6d3\"/\"default-dockercfg-9fh6r\"" Oct 30 01:22:24.515923 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:24.515899 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-e2d6d3\"/\"auth-test-model-e2d6d3-kserve-self-signed-certs\"" Oct 30 01:22:24.516131 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:24.515940 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-e2d6d3\"/\"kube-root-ca.crt\"" Oct 30 01:22:24.516131 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:24.516091 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-e2d6d3\"/\"openshift-service-ca.crt\"" Oct 30 01:22:24.616345 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:24.616312 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-model-cache\") pod \"auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps\" (UID: \"0957caa6-ff6b-4cf8-be88-717b6434a22f\") " pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" Oct 30 01:22:24.616555 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:24.616356 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0957caa6-ff6b-4cf8-be88-717b6434a22f-tls-certs\") pod \"auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps\" (UID: \"0957caa6-ff6b-4cf8-be88-717b6434a22f\") " pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" Oct 30 01:22:24.616555 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:24.616447 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-home\") pod \"auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps\" (UID: \"0957caa6-ff6b-4cf8-be88-717b6434a22f\") " pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" Oct 30 01:22:24.616691 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:24.616565 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hljx2\" (UniqueName: \"kubernetes.io/projected/0957caa6-ff6b-4cf8-be88-717b6434a22f-kube-api-access-hljx2\") pod \"auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps\" (UID: \"0957caa6-ff6b-4cf8-be88-717b6434a22f\") " pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" Oct 30 01:22:24.616691 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:24.616662 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-tmp-dir\") pod \"auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps\" (UID: \"0957caa6-ff6b-4cf8-be88-717b6434a22f\") " pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" Oct 30 01:22:24.616842 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:24.616780 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-dshm\") pod \"auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps\" (UID: \"0957caa6-ff6b-4cf8-be88-717b6434a22f\") " pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" Oct 30 01:22:24.616902 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:24.616853 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-model-cache\") pod \"auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps\" (UID: \"0957caa6-ff6b-4cf8-be88-717b6434a22f\") " pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" Oct 30 01:22:24.617238 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:24.617210 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-tmp-dir\") pod \"auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps\" (UID: \"0957caa6-ff6b-4cf8-be88-717b6434a22f\") " pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" Oct 30 01:22:24.617774 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:24.617232 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-home\") pod \"auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps\" (UID: \"0957caa6-ff6b-4cf8-be88-717b6434a22f\") " pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" Oct 30 01:22:24.621144 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:24.621122 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-dshm\") pod \"auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps\" (UID: \"0957caa6-ff6b-4cf8-be88-717b6434a22f\") " pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" Oct 30 01:22:24.627471 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:24.627440 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0957caa6-ff6b-4cf8-be88-717b6434a22f-tls-certs\") pod \"auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps\" (UID: \"0957caa6-ff6b-4cf8-be88-717b6434a22f\") " pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" Oct 30 01:22:24.641379 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:24.641349 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hljx2\" (UniqueName: \"kubernetes.io/projected/0957caa6-ff6b-4cf8-be88-717b6434a22f-kube-api-access-hljx2\") pod \"auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps\" (UID: \"0957caa6-ff6b-4cf8-be88-717b6434a22f\") " pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" Oct 30 01:22:24.804752 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:24.804590 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" Oct 30 01:22:25.215235 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:25.215142 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps"] Oct 30 01:22:25.722526 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:25.722428 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:25.806500 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:25.806464 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" event={"ID":"0957caa6-ff6b-4cf8-be88-717b6434a22f","Type":"ContainerStarted","Data":"d6c750959fc74b4fa37f00b4fde3105f8673d49cfef299b43c65b67810f134e2"} Oct 30 01:22:25.806696 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:25.806512 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" event={"ID":"0957caa6-ff6b-4cf8-be88-717b6434a22f","Type":"ContainerStarted","Data":"f53f1cc757035a227dc1b2473cd4d248bc4bd844db45f3b04def8872a162842a"} Oct 30 01:22:25.850705 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:25.850647 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" podStartSLOduration=1.850633029 podStartE2EDuration="1.850633029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:25.848806994 +0000 UTC m=+2346.164675320" watchObservedRunningTime="2026-10-30 01:22:25.850633029 +0000 UTC m=+2346.166501354" Oct 30 01:22:26.540187 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:26.540149 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:26.805575 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:26.805493 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" Oct 30 01:22:28.137987 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:28.137955 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:29.838544 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:29.838518 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:31.500371 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:31.500290 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:33.233927 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:33.233872 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:34.213852 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:34.213808 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:34.410119 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:34.410071 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:34.642441 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:34.642407 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:34.804883 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:34.804840 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" Oct 30 01:22:34.818567 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:34.818533 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" Oct 30 01:22:34.888136 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:34.888097 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" Oct 30 01:22:35.660022 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:35.659985 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:37.286530 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:37.286501 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:39.038838 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:39.038805 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:39.137985 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:39.137953 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:40.851215 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:40.851181 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:42.351070 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:42.351035 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:43.971505 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:43.971466 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:45.713676 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:45.713642 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:45.820950 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:45.820915 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:46.527380 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:46.527347 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:46.912229 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:46.912147 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:47.563476 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:47.563442 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr"] Oct 30 01:22:47.641676 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:47.641642 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:47.740840 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:47.740788 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:47.873651 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:47.873564 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:48.356639 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:48.356604 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:48.496399 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:48.496362 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:49.356404 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:49.356368 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:50.708871 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.708839 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb"] Oct 30 01:22:50.713217 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.713197 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.722978 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.722952 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8"] Oct 30 01:22:50.746203 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.746121 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:50.761536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.761503 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb"] Oct 30 01:22:50.784288 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.784256 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8"] Oct 30 01:22:50.791740 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.791710 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.791899 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.791861 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27a25c57-e305-43ad-9de6-e32231466d85-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.792106 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.792077 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.792163 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.792148 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.792201 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.792190 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27a25c57-e305-43ad-9de6-e32231466d85-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.792284 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.792264 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.792336 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.792320 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.792486 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.792468 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrcsk\" (UniqueName: \"kubernetes.io/projected/27a25c57-e305-43ad-9de6-e32231466d85-kube-api-access-nrcsk\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.792579 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.792565 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27a25c57-e305-43ad-9de6-e32231466d85-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.815418 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.815395 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-e92648-openshift-default-dockercfg-zt4vb\"" Oct 30 01:22:50.894745 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.894712 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/459d912e-ed51-4c50-a747-3a2767b30ee2-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:50.894958 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.894795 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrcsk\" (UniqueName: \"kubernetes.io/projected/27a25c57-e305-43ad-9de6-e32231466d85-kube-api-access-nrcsk\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.894958 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.894891 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:50.894958 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.894921 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27a25c57-e305-43ad-9de6-e32231466d85-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.894958 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.894956 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.895192 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.895055 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27a25c57-e305-43ad-9de6-e32231466d85-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.895192 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.895131 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:50.895312 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.895227 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:50.895312 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.895299 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:50.895404 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.895365 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.895445 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.895423 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw4qd\" (UniqueName: \"kubernetes.io/projected/459d912e-ed51-4c50-a747-3a2767b30ee2-kube-api-access-fw4qd\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:50.895504 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.895485 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:50.895552 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.895531 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.895611 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.895591 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27a25c57-e305-43ad-9de6-e32231466d85-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.895672 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.895600 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.895726 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.895671 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27a25c57-e305-43ad-9de6-e32231466d85-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.895726 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.895709 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:50.895880 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.895733 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.895880 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.895813 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.895880 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.895844 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.895880 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.895852 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:50.896112 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.896094 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.896157 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.896095 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.897952 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.897923 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.898473 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.898452 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27a25c57-e305-43ad-9de6-e32231466d85-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.917179 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.917150 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27a25c57-e305-43ad-9de6-e32231466d85-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.917437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.917416 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrcsk\" (UniqueName: \"kubernetes.io/projected/27a25c57-e305-43ad-9de6-e32231466d85-kube-api-access-nrcsk\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:50.951339 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.951310 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:50.997497 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.997400 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:50.997688 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.997539 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:50.997688 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.997584 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:50.997688 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.997618 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:50.997882 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.997688 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw4qd\" (UniqueName: \"kubernetes.io/projected/459d912e-ed51-4c50-a747-3a2767b30ee2-kube-api-access-fw4qd\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:50.997882 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.997744 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:50.997882 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.997857 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:50.998057 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.997933 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:50.998057 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.997998 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/459d912e-ed51-4c50-a747-3a2767b30ee2-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:50.998161 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.998100 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:50.998605 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.998572 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:50.998719 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.998618 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:50.999063 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.998715 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:50.999063 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:50.998788 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/459d912e-ed51-4c50-a747-3a2767b30ee2-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:51.000512 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:51.000487 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:51.001021 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:51.000943 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:51.020030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:51.019881 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:51.020224 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:51.020199 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw4qd\" (UniqueName: \"kubernetes.io/projected/459d912e-ed51-4c50-a747-3a2767b30ee2-kube-api-access-fw4qd\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:51.058658 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:51.058628 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:51.084542 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:51.084502 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:51.393196 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:51.393162 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb"] Oct 30 01:22:51.398120 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:51.398081 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:22:51.398245 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:51.398170 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:22:51.398245 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:51.398218 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:22:51.412348 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:51.412325 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8"] Oct 30 01:22:51.416380 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:22:51.416355 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459d912e_ed51_4c50_a747_3a2767b30ee2.slice/crio-cdc6bc54c979af78e4751fab521740c07a670e3e039df15ab6d8c20ae511fb62 WatchSource:0}: Error finding container cdc6bc54c979af78e4751fab521740c07a670e3e039df15ab6d8c20ae511fb62: Status 404 returned error can't find the container with id cdc6bc54c979af78e4751fab521740c07a670e3e039df15ab6d8c20ae511fb62 Oct 30 01:22:51.419709 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:51.419671 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:22:51.419829 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:51.419777 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:22:51.419829 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:51.419822 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:22:51.601049 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:22:51.601009 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:22:52.025707 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:52.025664 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" event={"ID":"459d912e-ed51-4c50-a747-3a2767b30ee2","Type":"ContainerStarted","Data":"1f7bd68f02bd6653d44459059816ccff225f9957c56721ce68211ec0c76dadd9"} Oct 30 01:22:52.026164 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:52.025732 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" event={"ID":"459d912e-ed51-4c50-a747-3a2767b30ee2","Type":"ContainerStarted","Data":"cdc6bc54c979af78e4751fab521740c07a670e3e039df15ab6d8c20ae511fb62"} Oct 30 01:22:52.027575 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:52.027549 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" event={"ID":"27a25c57-e305-43ad-9de6-e32231466d85","Type":"ContainerStarted","Data":"9940ffc6cb6a1001f4be5808d6862b58dcdfa3ebc2bebeafed145eb02c6c7f58"} Oct 30 01:22:52.027704 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:52.027599 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" event={"ID":"27a25c57-e305-43ad-9de6-e32231466d85","Type":"ContainerStarted","Data":"20f7d9154d2b3588584cb193103a06151913d603185dcf19f4adb4868158f7d9"} Oct 30 01:22:52.059577 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:52.059553 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:52.061649 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:52.061621 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 01:22:52.061785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:52.061674 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" podUID="27a25c57-e305-43ad-9de6-e32231466d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 01:22:52.076108 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:52.076049 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" podStartSLOduration=2.076036105 podStartE2EDuration="2.076036105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:52.071377553 +0000 UTC m=+2372.387245878" watchObservedRunningTime="2026-10-30 01:22:52.076036105 +0000 UTC m=+2372.391904450" Oct 30 01:22:52.085514 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:52.085490 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:52.086699 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:52.086676 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 01:22:52.086829 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:52.086727 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" podUID="459d912e-ed51-4c50-a747-3a2767b30ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 01:22:52.136964 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:52.134052 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" podStartSLOduration=2.134032022 podStartE2EDuration="2.134032022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:52.128690375 +0000 UTC m=+2372.444558723" watchObservedRunningTime="2026-10-30 01:22:52.134032022 +0000 UTC m=+2372.449900349" Oct 30 01:22:52.570790 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:52.565828 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr" podUID="adffe4ff-92a0-46de-93f0-e1f0a46da05d" containerName="payload-pre-processing" containerID="cri-o://31d6f878038a28754160021e18135b16ca02d2f109676353bbe5748f80cd8227" gracePeriod=25 Oct 30 01:22:52.572672 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:52.572277 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:52.885340 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:52.881297 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:53.102066 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:53.102026 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:53.102985 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:53.102955 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 01:22:53.103100 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:53.103014 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" podUID="459d912e-ed51-4c50-a747-3a2767b30ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 01:22:53.103575 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:53.103550 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 01:22:53.103656 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:53.103596 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" podUID="27a25c57-e305-43ad-9de6-e32231466d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 01:22:53.126848 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:53.117828 5300 generic.go:356] "Generic (PLEG): container finished" podID="adffe4ff-92a0-46de-93f0-e1f0a46da05d" containerID="31d6f878038a28754160021e18135b16ca02d2f109676353bbe5748f80cd8227" exitCode=0 Oct 30 01:22:53.128259 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:53.127307 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr" event={"ID":"adffe4ff-92a0-46de-93f0-e1f0a46da05d","Type":"ContainerDied","Data":"31d6f878038a28754160021e18135b16ca02d2f109676353bbe5748f80cd8227"} Oct 30 01:22:53.128259 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:53.127385 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr" event={"ID":"adffe4ff-92a0-46de-93f0-e1f0a46da05d","Type":"ContainerDied","Data":"cf59edb0c26480f1465af96c9bdc60a5cb7542e89ff08cee71a9d4f9dfa97734"} Oct 30 01:22:53.128259 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:53.127411 5300 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf59edb0c26480f1465af96c9bdc60a5cb7542e89ff08cee71a9d4f9dfa97734" Oct 30 01:22:53.138867 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:53.128674 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr" Oct 30 01:22:53.307883 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:53.307850 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adffe4ff-92a0-46de-93f0-e1f0a46da05d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adffe4ff-92a0-46de-93f0-e1f0a46da05d-plugins-config-volume\") pod \"adffe4ff-92a0-46de-93f0-e1f0a46da05d\" (UID: \"adffe4ff-92a0-46de-93f0-e1f0a46da05d\") " Oct 30 01:22:53.308064 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:53.308027 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adffe4ff-92a0-46de-93f0-e1f0a46da05d-kube-api-access-9gln9\" (UniqueName: \"kubernetes.io/projected/adffe4ff-92a0-46de-93f0-e1f0a46da05d-kube-api-access-9gln9\") pod \"adffe4ff-92a0-46de-93f0-e1f0a46da05d\" (UID: \"adffe4ff-92a0-46de-93f0-e1f0a46da05d\") " Oct 30 01:22:53.309718 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:53.309674 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adffe4ff-92a0-46de-93f0-e1f0a46da05d-plugins-config-volume" pod "adffe4ff-92a0-46de-93f0-e1f0a46da05d" (UID: "adffe4ff-92a0-46de-93f0-e1f0a46da05d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:22:53.329808 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:53.315329 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adffe4ff-92a0-46de-93f0-e1f0a46da05d-kube-api-access-9gln9" pod "adffe4ff-92a0-46de-93f0-e1f0a46da05d" (UID: "adffe4ff-92a0-46de-93f0-e1f0a46da05d"). InnerVolumeSpecName "kube-api-access-9gln9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:22:53.409906 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:53.409806 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gln9\" (UniqueName: \"kubernetes.io/projected/adffe4ff-92a0-46de-93f0-e1f0a46da05d-kube-api-access-9gln9\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:22:53.409906 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:53.409841 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adffe4ff-92a0-46de-93f0-e1f0a46da05d-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:22:54.070885 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.061200 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 01:22:54.070885 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.061263 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" podUID="27a25c57-e305-43ad-9de6-e32231466d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 01:22:54.085182 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.085139 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 01:22:54.085346 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.085211 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" podUID="459d912e-ed51-4c50-a747-3a2767b30ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 01:22:54.174200 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.172170 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr" Oct 30 01:22:54.278529 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.278421 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr"] Oct 30 01:22:54.284998 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.284966 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4be164-5676d98555-tgtpr"] Oct 30 01:22:54.470842 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.453546 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:54.470842 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.454515 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg"] Oct 30 01:22:54.490786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.480947 5300 cpu_manager.go:414] "containerMap: removing container" podUID="adffe4ff-92a0-46de-93f0-e1f0a46da05d" containerName="payload-pre-processing" Oct 30 01:22:54.490786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.480976 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adffe4ff-92a0-46de-93f0-e1f0a46da05d" containerName="payload-pre-processing" Oct 30 01:22:54.490786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.481186 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="adffe4ff-92a0-46de-93f0-e1f0a46da05d" containerName="payload-pre-processing" Oct 30 01:22:54.490786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.487135 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" Oct 30 01:22:54.495023 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.494996 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg"] Oct 30 01:22:54.505336 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.505311 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-d81f0a\"/\"kube-root-ca.crt\"" Oct 30 01:22:54.505481 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.505361 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-d81f0a\"/\"openshift-service-ca.crt\"" Oct 30 01:22:54.505536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.505488 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-d81f0a\"/\"auth-test-model-d81f0a-kserve-self-signed-certs\"" Oct 30 01:22:54.505596 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.505574 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-d81f0a\"/\"default-dockercfg-gzcr6\"" Oct 30 01:22:54.579491 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.579456 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:54.592284 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.592251 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91d3868-2251-4508-b4d2-0efb0265293a-tls-certs\") pod \"auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg\" (UID: \"d91d3868-2251-4508-b4d2-0efb0265293a\") " pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" Oct 30 01:22:54.592468 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.592332 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-dshm\") pod \"auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg\" (UID: \"d91d3868-2251-4508-b4d2-0efb0265293a\") " pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" Oct 30 01:22:54.592468 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.592414 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-model-cache\") pod \"auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg\" (UID: \"d91d3868-2251-4508-b4d2-0efb0265293a\") " pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" Oct 30 01:22:54.592602 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.592542 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-tmp-dir\") pod \"auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg\" (UID: \"d91d3868-2251-4508-b4d2-0efb0265293a\") " pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" Oct 30 01:22:54.592602 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.592576 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxz6g\" (UniqueName: \"kubernetes.io/projected/d91d3868-2251-4508-b4d2-0efb0265293a-kube-api-access-rxz6g\") pod \"auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg\" (UID: \"d91d3868-2251-4508-b4d2-0efb0265293a\") " pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" Oct 30 01:22:54.592723 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.592644 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-home\") pod \"auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg\" (UID: \"d91d3868-2251-4508-b4d2-0efb0265293a\") " pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" Oct 30 01:22:54.695524 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.695487 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91d3868-2251-4508-b4d2-0efb0265293a-tls-certs\") pod \"auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg\" (UID: \"d91d3868-2251-4508-b4d2-0efb0265293a\") " pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" Oct 30 01:22:54.695728 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.695562 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-dshm\") pod \"auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg\" (UID: \"d91d3868-2251-4508-b4d2-0efb0265293a\") " pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" Oct 30 01:22:54.695728 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.695627 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-model-cache\") pod \"auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg\" (UID: \"d91d3868-2251-4508-b4d2-0efb0265293a\") " pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" Oct 30 01:22:54.695728 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.695698 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-tmp-dir\") pod \"auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg\" (UID: \"d91d3868-2251-4508-b4d2-0efb0265293a\") " pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" Oct 30 01:22:54.695728 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.695723 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxz6g\" (UniqueName: \"kubernetes.io/projected/d91d3868-2251-4508-b4d2-0efb0265293a-kube-api-access-rxz6g\") pod \"auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg\" (UID: \"d91d3868-2251-4508-b4d2-0efb0265293a\") " pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" Oct 30 01:22:54.695973 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.695791 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-home\") pod \"auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg\" (UID: \"d91d3868-2251-4508-b4d2-0efb0265293a\") " pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" Oct 30 01:22:54.696302 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.696272 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-home\") pod \"auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg\" (UID: \"d91d3868-2251-4508-b4d2-0efb0265293a\") " pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" Oct 30 01:22:54.696416 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.696353 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-model-cache\") pod \"auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg\" (UID: \"d91d3868-2251-4508-b4d2-0efb0265293a\") " pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" Oct 30 01:22:54.696567 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.696545 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-tmp-dir\") pod \"auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg\" (UID: \"d91d3868-2251-4508-b4d2-0efb0265293a\") " pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" Oct 30 01:22:54.698540 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.698513 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-dshm\") pod \"auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg\" (UID: \"d91d3868-2251-4508-b4d2-0efb0265293a\") " pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" Oct 30 01:22:54.701158 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.701137 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91d3868-2251-4508-b4d2-0efb0265293a-tls-certs\") pod \"auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg\" (UID: \"d91d3868-2251-4508-b4d2-0efb0265293a\") " pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" Oct 30 01:22:54.759850 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.737005 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxz6g\" (UniqueName: \"kubernetes.io/projected/d91d3868-2251-4508-b4d2-0efb0265293a-kube-api-access-rxz6g\") pod \"auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg\" (UID: \"d91d3868-2251-4508-b4d2-0efb0265293a\") " pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" Oct 30 01:22:54.824783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:54.821165 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" Oct 30 01:22:55.041849 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:55.024544 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:55.128796 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:55.127530 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 01:22:55.128796 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:55.127602 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" podUID="27a25c57-e305-43ad-9de6-e32231466d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 01:22:55.163978 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:55.157406 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 01:22:55.163978 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:55.157482 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" podUID="459d912e-ed51-4c50-a747-3a2767b30ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 01:22:55.916802 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:55.888071 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adffe4ff-92a0-46de-93f0-e1f0a46da05d" path="/var/lib/kubelet/pods/adffe4ff-92a0-46de-93f0-e1f0a46da05d/volumes" Oct 30 01:22:55.936148 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:55.928435 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg"] Oct 30 01:22:56.095031 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:56.076414 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 01:22:56.095031 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:56.076467 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" podUID="27a25c57-e305-43ad-9de6-e32231466d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 01:22:56.095031 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:56.087902 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 01:22:56.095031 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:56.087960 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" podUID="459d912e-ed51-4c50-a747-3a2767b30ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 01:22:56.223323 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:56.223276 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" event={"ID":"d91d3868-2251-4508-b4d2-0efb0265293a","Type":"ContainerStarted","Data":"ead750cffc8d4211b185831c3bb8e4163fcb341d88edecf3d87bc708dd2ad957"} Oct 30 01:22:56.941787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:56.940572 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:57.059786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:57.059308 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 01:22:57.059786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:57.059371 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" podUID="27a25c57-e305-43ad-9de6-e32231466d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 01:22:57.092788 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:57.088783 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 01:22:57.092788 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:57.088864 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" podUID="459d912e-ed51-4c50-a747-3a2767b30ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 01:22:57.270698 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:57.270566 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" event={"ID":"d91d3868-2251-4508-b4d2-0efb0265293a","Type":"ContainerStarted","Data":"b0766915757611ae2424740bc3ad7c96376e7e06c1d19efc6d884641f5a034f5"} Oct 30 01:22:57.315752 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:57.315704 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" podStartSLOduration=3.315689232 podStartE2EDuration="3.315689232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:57.314302759 +0000 UTC m=+2377.630171083" watchObservedRunningTime="2026-10-30 01:22:57.315689232 +0000 UTC m=+2377.631557558" Oct 30 01:22:57.873726 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:57.873677 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" Oct 30 01:22:58.059539 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:58.059490 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 01:22:58.060114 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:58.059561 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" podUID="27a25c57-e305-43ad-9de6-e32231466d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 01:22:58.085589 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:58.085554 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 01:22:58.085836 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:58.085608 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" podUID="459d912e-ed51-4c50-a747-3a2767b30ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 01:22:58.761010 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:58.760977 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:22:59.064521 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:59.064442 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:59.065032 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:59.065015 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:59.065800 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:59.065747 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:22:59.090415 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:59.090387 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:59.090675 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:59.090657 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:22:59.091456 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:22:59.091438 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:23:00.425470 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:00.425429 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:01.896521 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:01.896485 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:03.644242 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:03.644206 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:04.825446 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:04.825413 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" Oct 30 01:23:04.838614 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:04.838588 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" Oct 30 01:23:05.232655 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:05.232625 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:05.351296 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:05.351265 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" Oct 30 01:23:06.514114 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:06.514082 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:07.056070 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:07.056031 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:08.184946 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:08.184913 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:08.663512 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:08.663478 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:10.272549 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:10.272513 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:11.597966 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.597930 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts"] Oct 30 01:23:11.603513 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.603487 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" Oct 30 01:23:11.620022 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.619993 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts"] Oct 30 01:23:11.628566 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.628537 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-999ff9\"/\"kube-root-ca.crt\"" Oct 30 01:23:11.628730 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.628646 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-999ff9\"/\"openshift-service-ca.crt\"" Oct 30 01:23:11.636629 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.633899 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-999ff9\"/\"auth-test-model-999ff9-kserve-self-signed-certs\"" Oct 30 01:23:11.636629 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.633988 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-999ff9\"/\"default-dockercfg-q6lfb\"" Oct 30 01:23:11.636629 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.636266 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:11.700827 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.700742 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-tmp-dir\") pod \"auth-test-model-999ff9-kserve-6958bb4d77-5rvts\" (UID: \"bb602fa6-731e-4991-9504-7c743abb828b\") " pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" Oct 30 01:23:11.700998 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.700857 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-dshm\") pod \"auth-test-model-999ff9-kserve-6958bb4d77-5rvts\" (UID: \"bb602fa6-731e-4991-9504-7c743abb828b\") " pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" Oct 30 01:23:11.700998 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.700987 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqgr4\" (UniqueName: \"kubernetes.io/projected/bb602fa6-731e-4991-9504-7c743abb828b-kube-api-access-bqgr4\") pod \"auth-test-model-999ff9-kserve-6958bb4d77-5rvts\" (UID: \"bb602fa6-731e-4991-9504-7c743abb828b\") " pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" Oct 30 01:23:11.701079 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.701054 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb602fa6-731e-4991-9504-7c743abb828b-tls-certs\") pod \"auth-test-model-999ff9-kserve-6958bb4d77-5rvts\" (UID: \"bb602fa6-731e-4991-9504-7c743abb828b\") " pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" Oct 30 01:23:11.701211 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.701188 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-home\") pod \"auth-test-model-999ff9-kserve-6958bb4d77-5rvts\" (UID: \"bb602fa6-731e-4991-9504-7c743abb828b\") " pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" Oct 30 01:23:11.701281 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.701234 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-model-cache\") pod \"auth-test-model-999ff9-kserve-6958bb4d77-5rvts\" (UID: \"bb602fa6-731e-4991-9504-7c743abb828b\") " pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" Oct 30 01:23:11.802462 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.802422 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqgr4\" (UniqueName: \"kubernetes.io/projected/bb602fa6-731e-4991-9504-7c743abb828b-kube-api-access-bqgr4\") pod \"auth-test-model-999ff9-kserve-6958bb4d77-5rvts\" (UID: \"bb602fa6-731e-4991-9504-7c743abb828b\") " pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" Oct 30 01:23:11.802638 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.802515 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb602fa6-731e-4991-9504-7c743abb828b-tls-certs\") pod \"auth-test-model-999ff9-kserve-6958bb4d77-5rvts\" (UID: \"bb602fa6-731e-4991-9504-7c743abb828b\") " pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" Oct 30 01:23:11.802638 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.802584 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-home\") pod \"auth-test-model-999ff9-kserve-6958bb4d77-5rvts\" (UID: \"bb602fa6-731e-4991-9504-7c743abb828b\") " pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" Oct 30 01:23:11.802823 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.802642 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-model-cache\") pod \"auth-test-model-999ff9-kserve-6958bb4d77-5rvts\" (UID: \"bb602fa6-731e-4991-9504-7c743abb828b\") " pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" Oct 30 01:23:11.802930 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.802895 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-tmp-dir\") pod \"auth-test-model-999ff9-kserve-6958bb4d77-5rvts\" (UID: \"bb602fa6-731e-4991-9504-7c743abb828b\") " pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" Oct 30 01:23:11.803077 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.802949 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-dshm\") pod \"auth-test-model-999ff9-kserve-6958bb4d77-5rvts\" (UID: \"bb602fa6-731e-4991-9504-7c743abb828b\") " pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" Oct 30 01:23:11.803077 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.803046 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-model-cache\") pod \"auth-test-model-999ff9-kserve-6958bb4d77-5rvts\" (UID: \"bb602fa6-731e-4991-9504-7c743abb828b\") " pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" Oct 30 01:23:11.803179 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.803075 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-home\") pod \"auth-test-model-999ff9-kserve-6958bb4d77-5rvts\" (UID: \"bb602fa6-731e-4991-9504-7c743abb828b\") " pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" Oct 30 01:23:11.803331 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.803281 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-tmp-dir\") pod \"auth-test-model-999ff9-kserve-6958bb4d77-5rvts\" (UID: \"bb602fa6-731e-4991-9504-7c743abb828b\") " pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" Oct 30 01:23:11.805711 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.805416 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-dshm\") pod \"auth-test-model-999ff9-kserve-6958bb4d77-5rvts\" (UID: \"bb602fa6-731e-4991-9504-7c743abb828b\") " pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" Oct 30 01:23:11.805711 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.805641 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb602fa6-731e-4991-9504-7c743abb828b-tls-certs\") pod \"auth-test-model-999ff9-kserve-6958bb4d77-5rvts\" (UID: \"bb602fa6-731e-4991-9504-7c743abb828b\") " pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" Oct 30 01:23:11.841737 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.840621 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqgr4\" (UniqueName: \"kubernetes.io/projected/bb602fa6-731e-4991-9504-7c743abb828b-kube-api-access-bqgr4\") pod \"auth-test-model-999ff9-kserve-6958bb4d77-5rvts\" (UID: \"bb602fa6-731e-4991-9504-7c743abb828b\") " pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" Oct 30 01:23:11.933835 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.933689 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" Oct 30 01:23:11.976142 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:11.976109 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:12.256676 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:12.255421 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts"] Oct 30 01:23:12.261724 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:23:12.261692 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb602fa6_731e_4991_9504_7c743abb828b.slice/crio-1e5c606bc84bba5a60866004336f7d4b929634449aef38ca8044e865a363051f WatchSource:0}: Error finding container 1e5c606bc84bba5a60866004336f7d4b929634449aef38ca8044e865a363051f: Status 404 returned error can't find the container with id 1e5c606bc84bba5a60866004336f7d4b929634449aef38ca8044e865a363051f Oct 30 01:23:12.406784 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:12.404624 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" event={"ID":"bb602fa6-731e-4991-9504-7c743abb828b","Type":"ContainerStarted","Data":"f0052e01936627d657d31b054e2ee386a695f5f1293e3011e6b99fa52acfbf2f"} Oct 30 01:23:12.406784 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:12.404666 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" event={"ID":"bb602fa6-731e-4991-9504-7c743abb828b","Type":"ContainerStarted","Data":"1e5c606bc84bba5a60866004336f7d4b929634449aef38ca8044e865a363051f"} Oct 30 01:23:12.449954 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:12.449880 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" podStartSLOduration=1.449861108 podStartE2EDuration="1.449861108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:12.446745457 +0000 UTC m=+2392.762613790" watchObservedRunningTime="2026-10-30 01:23:12.449861108 +0000 UTC m=+2392.765729433" Oct 30 01:23:12.934148 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:12.934120 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" Oct 30 01:23:13.509122 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:13.509087 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:14.429973 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:14.429935 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25"] Oct 30 01:23:14.476319 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:14.476284 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:15.038041 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:15.038008 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:15.312521 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:15.312431 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:15.798425 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:15.798397 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:15.942510 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:15.942474 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:16.760124 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:16.760084 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:18.738061 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:18.738025 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:19.823480 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:19.823447 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:19.926217 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:19.926183 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:20.039980 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:20.039947 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:20.541807 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:20.541740 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:20.641873 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:20.641835 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:21.085491 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:21.085456 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:21.191507 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:21.191475 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:21.597699 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:23:21.597656 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:23:21.934369 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:21.934286 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" Oct 30 01:23:21.948024 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:21.947997 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" Oct 30 01:23:22.131647 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:22.131611 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:22.319218 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:22.319187 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:22.416449 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:22.416419 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:22.504003 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:22.503975 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" Oct 30 01:23:22.519308 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:22.519281 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb"] Oct 30 01:23:22.525799 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:22.525735 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb" Oct 30 01:23:22.542990 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:22.542962 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-73831d\"" Oct 30 01:23:22.543120 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:22.543013 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-73831d-dockercfg-c269j\"" Oct 30 01:23:22.545083 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:22.545060 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb"] Oct 30 01:23:22.638178 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:22.629973 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk4vl\" (UniqueName: \"kubernetes.io/projected/aced2f61-18a5-4919-88a5-a76f6952085d-kube-api-access-dk4vl\") pod \"payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb\" (UID: \"aced2f61-18a5-4919-88a5-a76f6952085d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb" Oct 30 01:23:22.638990 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:22.638953 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aced2f61-18a5-4919-88a5-a76f6952085d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb\" (UID: \"aced2f61-18a5-4919-88a5-a76f6952085d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb" Oct 30 01:23:22.759991 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:22.759958 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aced2f61-18a5-4919-88a5-a76f6952085d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb\" (UID: \"aced2f61-18a5-4919-88a5-a76f6952085d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb" Oct 30 01:23:22.760263 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:22.760240 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk4vl\" (UniqueName: \"kubernetes.io/projected/aced2f61-18a5-4919-88a5-a76f6952085d-kube-api-access-dk4vl\") pod \"payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb\" (UID: \"aced2f61-18a5-4919-88a5-a76f6952085d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb" Oct 30 01:23:22.761461 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:22.761433 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aced2f61-18a5-4919-88a5-a76f6952085d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb\" (UID: \"aced2f61-18a5-4919-88a5-a76f6952085d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb" Oct 30 01:23:22.792580 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:22.792547 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk4vl\" (UniqueName: \"kubernetes.io/projected/aced2f61-18a5-4919-88a5-a76f6952085d-kube-api-access-dk4vl\") pod \"payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb\" (UID: \"aced2f61-18a5-4919-88a5-a76f6952085d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb" Oct 30 01:23:22.840207 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:22.840176 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb" Oct 30 01:23:23.153624 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:23.153553 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb"] Oct 30 01:23:23.153990 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:23:23.153679 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaced2f61_18a5_4919_88a5_a76f6952085d.slice/crio-2590ee02b5c229a7a0eac913ed2b4117086d00aa1aafb44629b91adf5d204930 WatchSource:0}: Error finding container 2590ee02b5c229a7a0eac913ed2b4117086d00aa1aafb44629b91adf5d204930: Status 404 returned error can't find the container with id 2590ee02b5c229a7a0eac913ed2b4117086d00aa1aafb44629b91adf5d204930 Oct 30 01:23:23.502739 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:23.502700 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb" event={"ID":"aced2f61-18a5-4919-88a5-a76f6952085d","Type":"ContainerStarted","Data":"c23b023458ccdad8d53dc0086f79abb6aa5407fefcb17c7ca89ac4ab639663e2"} Oct 30 01:23:23.503041 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:23.502781 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb" event={"ID":"aced2f61-18a5-4919-88a5-a76f6952085d","Type":"ContainerStarted","Data":"2590ee02b5c229a7a0eac913ed2b4117086d00aa1aafb44629b91adf5d204930"} Oct 30 01:23:23.550048 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:23.549995 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb" podStartSLOduration=1.549979639 podStartE2EDuration="1.549979639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:23.546896336 +0000 UTC m=+2403.862764674" watchObservedRunningTime="2026-10-30 01:23:23.549979639 +0000 UTC m=+2403.865848016" Oct 30 01:23:23.764722 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:23.764568 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:24.510950 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:24.510915 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb" Oct 30 01:23:25.094439 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:25.094389 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:25.257603 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:25.257573 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:25.885858 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:25.885806 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:26.836859 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:26.836755 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:26.937144 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:26.937108 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:27.674057 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:27.673600 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:28.975939 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:28.975906 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:29.231640 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:29.213134 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:29.506849 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:29.472892 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" podUID="2c040a91-8aa7-4fd9-86ef-b9e38db39f6f" containerName="main" containerID="cri-o://8e79610fea6373f5170e50c87e7b9f7ec69a0757f6ab635902da1d2d6fd00b82" gracePeriod=45 Oct 30 01:23:30.063150 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.063113 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" Oct 30 01:23:30.133455 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.133362 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-dshm\") pod \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\" (UID: \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\") " Oct 30 01:23:30.133455 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.133401 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-model-cache\") pod \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\" (UID: \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\") " Oct 30 01:23:30.133710 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.133462 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-tmp-dir\") pod \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\" (UID: \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\") " Oct 30 01:23:30.133828 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.133729 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-model-cache" pod "2c040a91-8aa7-4fd9-86ef-b9e38db39f6f" (UID: "2c040a91-8aa7-4fd9-86ef-b9e38db39f6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:30.133828 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.133740 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-tmp-dir" pod "2c040a91-8aa7-4fd9-86ef-b9e38db39f6f" (UID: "2c040a91-8aa7-4fd9-86ef-b9e38db39f6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:30.133968 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.133887 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-tls-certs\") pod \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\" (UID: \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\") " Oct 30 01:23:30.134045 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.134026 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-kube-api-access-987vg\" (UniqueName: \"kubernetes.io/projected/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-kube-api-access-987vg\") pod \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\" (UID: \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\") " Oct 30 01:23:30.134122 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.134106 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-home\" (UniqueName: \"kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-home\") pod \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\" (UID: \"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f\") " Oct 30 01:23:30.134536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.134508 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-home" pod "2c040a91-8aa7-4fd9-86ef-b9e38db39f6f" (UID: "2c040a91-8aa7-4fd9-86ef-b9e38db39f6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:30.134969 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.134948 5300 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-model-cache\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:23:30.134969 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.134970 5300 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-tmp-dir\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:23:30.135127 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.134978 5300 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-home\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:23:30.136803 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.136774 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-tls-certs" pod "2c040a91-8aa7-4fd9-86ef-b9e38db39f6f" (UID: "2c040a91-8aa7-4fd9-86ef-b9e38db39f6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:30.136883 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.136817 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-dshm" pod "2c040a91-8aa7-4fd9-86ef-b9e38db39f6f" (UID: "2c040a91-8aa7-4fd9-86ef-b9e38db39f6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:30.137131 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.137105 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-kube-api-access-987vg" pod "2c040a91-8aa7-4fd9-86ef-b9e38db39f6f" (UID: "2c040a91-8aa7-4fd9-86ef-b9e38db39f6f"). InnerVolumeSpecName "kube-api-access-987vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:30.163397 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.163362 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:30.238614 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.238319 5300 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-dshm\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:23:30.238614 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.238359 5300 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-tls-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:23:30.238614 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.238376 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-987vg\" (UniqueName: \"kubernetes.io/projected/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f-kube-api-access-987vg\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:23:30.626721 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.626675 5300 generic.go:356] "Generic (PLEG): container finished" podID="2c040a91-8aa7-4fd9-86ef-b9e38db39f6f" containerID="8e79610fea6373f5170e50c87e7b9f7ec69a0757f6ab635902da1d2d6fd00b82" exitCode=0 Oct 30 01:23:30.626938 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.626746 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" Oct 30 01:23:30.626938 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.626879 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" event={"ID":"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f","Type":"ContainerDied","Data":"8e79610fea6373f5170e50c87e7b9f7ec69a0757f6ab635902da1d2d6fd00b82"} Oct 30 01:23:30.626938 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.626914 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25" event={"ID":"2c040a91-8aa7-4fd9-86ef-b9e38db39f6f","Type":"ContainerDied","Data":"a3ca1e847fffdc014176c75048f2ec93b97a1cccaf234554ab3af5a311c8f745"} Oct 30 01:23:30.626938 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.626931 5300 scope.go:122] "RemoveContainer" containerID="8e79610fea6373f5170e50c87e7b9f7ec69a0757f6ab635902da1d2d6fd00b82" Oct 30 01:23:30.641161 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.641135 5300 scope.go:122] "RemoveContainer" containerID="8e79610fea6373f5170e50c87e7b9f7ec69a0757f6ab635902da1d2d6fd00b82" Oct 30 01:23:30.641488 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:23:30.641464 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e79610fea6373f5170e50c87e7b9f7ec69a0757f6ab635902da1d2d6fd00b82\": container with ID starting with 8e79610fea6373f5170e50c87e7b9f7ec69a0757f6ab635902da1d2d6fd00b82 not found: ID does not exist" containerID="8e79610fea6373f5170e50c87e7b9f7ec69a0757f6ab635902da1d2d6fd00b82" Oct 30 01:23:30.641551 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.641496 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e79610fea6373f5170e50c87e7b9f7ec69a0757f6ab635902da1d2d6fd00b82"} err="failed to get container status \"8e79610fea6373f5170e50c87e7b9f7ec69a0757f6ab635902da1d2d6fd00b82\": rpc error: code = NotFound desc = could not find container \"8e79610fea6373f5170e50c87e7b9f7ec69a0757f6ab635902da1d2d6fd00b82\": container with ID starting with 8e79610fea6373f5170e50c87e7b9f7ec69a0757f6ab635902da1d2d6fd00b82 not found: ID does not exist" Oct 30 01:23:30.687100 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.687069 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25"] Oct 30 01:23:30.700988 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:30.700943 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-2be695/auth-test-model-2be695-kserve-797888bdbf-z8m25"] Oct 30 01:23:31.827667 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:31.827632 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c040a91-8aa7-4fd9-86ef-b9e38db39f6f" path="/var/lib/kubelet/pods/2c040a91-8aa7-4fd9-86ef-b9e38db39f6f/volumes" Oct 30 01:23:31.935911 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:31.935876 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:33.823835 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:33.823802 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:34.498027 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:34.497991 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:35.419356 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:35.419317 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:35.540150 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:35.540120 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb" Oct 30 01:23:35.832552 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:35.832521 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6"] Oct 30 01:23:35.833490 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:35.833467 5300 cpu_manager.go:414] "containerMap: removing container" podUID="2c040a91-8aa7-4fd9-86ef-b9e38db39f6f" containerName="main" Oct 30 01:23:35.833490 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:35.833487 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c040a91-8aa7-4fd9-86ef-b9e38db39f6f" containerName="main" Oct 30 01:23:35.833689 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:35.833615 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c040a91-8aa7-4fd9-86ef-b9e38db39f6f" containerName="main" Oct 30 01:23:35.838084 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:35.838063 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:35.862319 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:35.861932 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6"] Oct 30 01:23:35.949756 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:35.949720 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:35.949975 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:35.949857 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k2lq\" (UniqueName: \"kubernetes.io/projected/1e2be936-3c18-4d9b-87a4-800cb91748b7-kube-api-access-7k2lq\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:35.949975 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:35.949951 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:35.950067 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:35.950045 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e2be936-3c18-4d9b-87a4-800cb91748b7-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:35.950140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:35.950119 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:35.950347 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:35.950332 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:35.950382 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:35.950369 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:35.950552 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:35.950535 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:35.950594 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:35.950581 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:36.052108 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.052074 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:36.052316 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.052123 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:36.052316 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.052211 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:36.052316 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.052254 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:36.052316 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.052306 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:36.052563 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.052350 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k2lq\" (UniqueName: \"kubernetes.io/projected/1e2be936-3c18-4d9b-87a4-800cb91748b7-kube-api-access-7k2lq\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:36.052563 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.052448 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:36.052563 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.052494 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e2be936-3c18-4d9b-87a4-800cb91748b7-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:36.052563 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.052531 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:36.052798 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.052629 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-workload-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:36.052798 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.052645 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-credential-socket\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:36.052798 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.052731 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-data\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:36.052798 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.052742 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-workload-certs\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:36.053396 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.053375 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e2be936-3c18-4d9b-87a4-800cb91748b7-istiod-ca-cert\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:36.054941 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.054918 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-envoy\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:36.055197 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.055178 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-podinfo\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:36.073523 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.073494 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-token\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:36.073614 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.073566 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k2lq\" (UniqueName: \"kubernetes.io/projected/1e2be936-3c18-4d9b-87a4-800cb91748b7-kube-api-access-7k2lq\") pod \"e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:36.160015 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.159929 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:36.498843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.498813 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6"] Oct 30 01:23:36.499541 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:23:36.499513 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e2be936_3c18_4d9b_87a4_800cb91748b7.slice/crio-61d4fb29919b310b3022182d1dab831385a71a2fa8c6d11b596c74218d5dcede WatchSource:0}: Error finding container 61d4fb29919b310b3022182d1dab831385a71a2fa8c6d11b596c74218d5dcede: Status 404 returned error can't find the container with id 61d4fb29919b310b3022182d1dab831385a71a2fa8c6d11b596c74218d5dcede Oct 30 01:23:36.502743 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.502704 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:23:36.502847 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.502799 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:23:36.502847 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.502838 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:23:36.697405 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.697367 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" event={"ID":"1e2be936-3c18-4d9b-87a4-800cb91748b7","Type":"ContainerStarted","Data":"56ec838d06e5d7f9ef6b0da56927c45ef9cdf63ae351ac3ff01c011132cef27f"} Oct 30 01:23:36.697405 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.697423 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" event={"ID":"1e2be936-3c18-4d9b-87a4-800cb91748b7","Type":"ContainerStarted","Data":"61d4fb29919b310b3022182d1dab831385a71a2fa8c6d11b596c74218d5dcede"} Oct 30 01:23:36.748193 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.748086 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" podStartSLOduration=1.748069538 podStartE2EDuration="1.748069538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:36.745136386 +0000 UTC m=+2417.061004713" watchObservedRunningTime="2026-10-30 01:23:36.748069538 +0000 UTC m=+2417.063937860" Oct 30 01:23:36.918879 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.918849 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c"] Oct 30 01:23:36.937529 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.937490 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c" Oct 30 01:23:36.947653 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.947626 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c"] Oct 30 01:23:36.954971 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:36.954926 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-7f0263-serving-cert\"" Oct 30 01:23:37.014568 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:37.014449 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:37.069874 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:37.069838 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c\" (UID: \"5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c" Oct 30 01:23:37.070035 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:37.069893 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c\" (UID: \"5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c" Oct 30 01:23:37.070083 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:37.070060 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mncbk\" (UniqueName: \"kubernetes.io/projected/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-kube-api-access-mncbk\") pod \"maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c\" (UID: \"5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c" Oct 30 01:23:37.070252 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:37.070233 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-metrics-certs\") pod \"maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c\" (UID: \"5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c" Oct 30 01:23:37.125131 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:37.125095 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:37.161028 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:37.160991 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:37.162593 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:37.162570 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 01:23:37.162681 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:37.162618 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" podUID="1e2be936-3c18-4d9b-87a4-800cb91748b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 01:23:37.171858 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:37.171827 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c\" (UID: \"5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c" Oct 30 01:23:37.172018 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:37.171879 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c\" (UID: \"5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c" Oct 30 01:23:37.172018 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:37.171947 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mncbk\" (UniqueName: \"kubernetes.io/projected/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-kube-api-access-mncbk\") pod \"maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c\" (UID: \"5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c" Oct 30 01:23:37.172145 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:37.172100 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-metrics-certs\") pod \"maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c\" (UID: \"5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c" Oct 30 01:23:37.172669 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:37.172638 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c\" (UID: \"5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c" Oct 30 01:23:37.174974 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:37.174946 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c\" (UID: \"5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c" Oct 30 01:23:37.175474 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:37.175446 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-metrics-certs\") pod \"maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c\" (UID: \"5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c" Oct 30 01:23:37.192868 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:37.192842 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mncbk\" (UniqueName: \"kubernetes.io/projected/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-kube-api-access-mncbk\") pod \"maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c\" (UID: \"5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c" Oct 30 01:23:37.223513 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:37.223481 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:37.263946 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:37.263900 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c" Oct 30 01:23:37.365889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:37.365856 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:37.588983 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:23:37.588955 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0eb1e1_8d79_4ace_a2d1_f1f5383a3001.slice/crio-290fae8e46f84e151a239b25214c2ba58d9b0621bdd3d45b081211d059ae7f34 WatchSource:0}: Error finding container 290fae8e46f84e151a239b25214c2ba58d9b0621bdd3d45b081211d059ae7f34: Status 404 returned error can't find the container with id 290fae8e46f84e151a239b25214c2ba58d9b0621bdd3d45b081211d059ae7f34 Oct 30 01:23:37.589856 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:37.589831 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c"] Oct 30 01:23:37.710215 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:37.710176 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c" event={"ID":"5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001","Type":"ContainerStarted","Data":"290fae8e46f84e151a239b25214c2ba58d9b0621bdd3d45b081211d059ae7f34"} Oct 30 01:23:37.899375 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:37.899290 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:37.999168 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:37.999143 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:38.160343 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:38.160253 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 01:23:38.160343 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:38.160317 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" podUID="1e2be936-3c18-4d9b-87a4-800cb91748b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 01:23:38.747184 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:38.747135 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c" event={"ID":"5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001","Type":"ContainerStarted","Data":"134ef7a3e1a73f5fbdeadf539f5bf587d59da39f77557d6adfe7ff785db1ebb0"} Oct 30 01:23:38.747716 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:38.747599 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c" Oct 30 01:23:38.811313 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:38.811250 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c" podStartSLOduration=2.42579911 podStartE2EDuration="2.811233676s" totalImagesPullingTime="385.434566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:38.810121717 +0000 UTC m=+2419.125990041" watchObservedRunningTime="2026-10-30 01:23:38.811233676 +0000 UTC m=+2419.127102000" Oct 30 01:23:38.839781 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:38.839586 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:39.175843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:39.163183 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 01:23:39.175843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:39.163246 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" podUID="1e2be936-3c18-4d9b-87a4-800cb91748b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 01:23:39.972004 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:39.971941 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8"] Oct 30 01:23:39.972637 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:39.972610 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:39.980234 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:39.980210 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" Oct 30 01:23:39.990340 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:39.987978 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8"] Oct 30 01:23:39.998417 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:39.998395 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-e92648\"/\"default-dockercfg-rktlq\"" Oct 30 01:23:39.998908 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:39.998886 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-e92648\"/\"auth-test-model-e92648-kserve-self-signed-certs\"" Oct 30 01:23:40.006435 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:40.006410 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf9h9\" (UniqueName: \"kubernetes.io/projected/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-kube-api-access-jf9h9\") pod \"auth-test-model-e92648-kserve-549f496b8f-hkmw8\" (UID: \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\") " pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" Oct 30 01:23:40.006532 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:40.006517 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-model-cache\") pod \"auth-test-model-e92648-kserve-549f496b8f-hkmw8\" (UID: \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\") " pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" Oct 30 01:23:40.007500 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:40.007481 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-dshm\") pod \"auth-test-model-e92648-kserve-549f496b8f-hkmw8\" (UID: \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\") " pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" Oct 30 01:23:40.007586 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:40.007557 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-home\") pod \"auth-test-model-e92648-kserve-549f496b8f-hkmw8\" (UID: \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\") " pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" Oct 30 01:23:40.007657 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:40.007642 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-tmp-dir\") pod \"auth-test-model-e92648-kserve-549f496b8f-hkmw8\" (UID: \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\") " pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" Oct 30 01:23:40.007836 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:40.007818 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-tls-certs\") pod \"auth-test-model-e92648-kserve-549f496b8f-hkmw8\" (UID: \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\") " pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" Oct 30 01:23:40.012515 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:40.012496 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-e92648\"/\"openshift-service-ca.crt\"" Oct 30 01:23:40.012609 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:40.012517 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-e92648\"/\"kube-root-ca.crt\"" Oct 30 01:23:40.108811 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:40.108776 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-dshm\") pod \"auth-test-model-e92648-kserve-549f496b8f-hkmw8\" (UID: \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\") " pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" Oct 30 01:23:40.108998 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:40.108854 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-home\") pod \"auth-test-model-e92648-kserve-549f496b8f-hkmw8\" (UID: \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\") " pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" Oct 30 01:23:40.108998 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:40.108936 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-tmp-dir\") pod \"auth-test-model-e92648-kserve-549f496b8f-hkmw8\" (UID: \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\") " pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" Oct 30 01:23:40.108998 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:40.108956 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-tls-certs\") pod \"auth-test-model-e92648-kserve-549f496b8f-hkmw8\" (UID: \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\") " pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" Oct 30 01:23:40.109143 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:40.109024 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf9h9\" (UniqueName: \"kubernetes.io/projected/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-kube-api-access-jf9h9\") pod \"auth-test-model-e92648-kserve-549f496b8f-hkmw8\" (UID: \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\") " pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" Oct 30 01:23:40.109143 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:40.109099 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-model-cache\") pod \"auth-test-model-e92648-kserve-549f496b8f-hkmw8\" (UID: \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\") " pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" Oct 30 01:23:40.109313 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:40.109252 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-home\") pod \"auth-test-model-e92648-kserve-549f496b8f-hkmw8\" (UID: \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\") " pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" Oct 30 01:23:40.109370 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:40.109320 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-tmp-dir\") pod \"auth-test-model-e92648-kserve-549f496b8f-hkmw8\" (UID: \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\") " pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" Oct 30 01:23:40.109474 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:40.109456 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-model-cache\") pod \"auth-test-model-e92648-kserve-549f496b8f-hkmw8\" (UID: \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\") " pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" Oct 30 01:23:40.111124 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:40.111102 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-dshm\") pod \"auth-test-model-e92648-kserve-549f496b8f-hkmw8\" (UID: \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\") " pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" Oct 30 01:23:40.111456 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:40.111437 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-tls-certs\") pod \"auth-test-model-e92648-kserve-549f496b8f-hkmw8\" (UID: \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\") " pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" Oct 30 01:23:40.149911 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:40.149870 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf9h9\" (UniqueName: \"kubernetes.io/projected/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-kube-api-access-jf9h9\") pod \"auth-test-model-e92648-kserve-549f496b8f-hkmw8\" (UID: \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\") " pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" Oct 30 01:23:40.171104 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:40.168904 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 01:23:40.171104 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:40.168977 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" podUID="1e2be936-3c18-4d9b-87a4-800cb91748b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 01:23:40.304662 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:40.304566 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" Oct 30 01:23:40.698998 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:40.698917 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:40.703429 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:40.703403 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8"] Oct 30 01:23:40.703564 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:23:40.703543 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e04fb5b_dba1_4e40_8a14_ff555398e2ee.slice/crio-756db2bf48e1eae74e63d7d95074b36e8e1330c6bbf6ac58f0de8c56f5abb359 WatchSource:0}: Error finding container 756db2bf48e1eae74e63d7d95074b36e8e1330c6bbf6ac58f0de8c56f5abb359: Status 404 returned error can't find the container with id 756db2bf48e1eae74e63d7d95074b36e8e1330c6bbf6ac58f0de8c56f5abb359 Oct 30 01:23:40.778332 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:40.776851 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" event={"ID":"6e04fb5b-dba1-4e40-8a14-ff555398e2ee","Type":"ContainerStarted","Data":"756db2bf48e1eae74e63d7d95074b36e8e1330c6bbf6ac58f0de8c56f5abb359"} Oct 30 01:23:41.160475 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:41.160442 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 01:23:41.160859 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:41.160497 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" podUID="1e2be936-3c18-4d9b-87a4-800cb91748b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 01:23:41.786384 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:41.786344 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" event={"ID":"6e04fb5b-dba1-4e40-8a14-ff555398e2ee","Type":"ContainerStarted","Data":"ec4c1dbecae5d8d2241248eb006dda85dbd880e34aee7c759155dc410d5aae96"} Oct 30 01:23:41.851809 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:41.849183 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" podStartSLOduration=2.849165182 podStartE2EDuration="2.849165182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:41.848047055 +0000 UTC m=+2422.163915408" watchObservedRunningTime="2026-10-30 01:23:41.849165182 +0000 UTC m=+2422.165033508" Oct 30 01:23:42.160682 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:42.160533 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 01:23:42.160682 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:42.160603 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" podUID="1e2be936-3c18-4d9b-87a4-800cb91748b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 01:23:42.270981 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:42.270936 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:42.308245 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:42.305619 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" Oct 30 01:23:43.160331 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:43.160279 5300 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 01:23:43.160521 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:43.160339 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" podUID="1e2be936-3c18-4d9b-87a4-800cb91748b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 01:23:43.842644 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:43.842612 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:44.166185 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:44.166094 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:44.166533 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:44.166514 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:44.167243 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:44.167224 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:23:44.763976 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:44.763946 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c" Oct 30 01:23:45.462979 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.462949 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666"] Oct 30 01:23:45.467871 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.467847 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.491418 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.491390 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666"] Oct 30 01:23:45.535173 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.535141 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-credential-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.535351 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.535204 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-envoy\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.535351 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.535311 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istiod-ca-cert\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.535432 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.535393 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-workload-certs\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.535478 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.535462 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-token\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.535519 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.535501 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-workload-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.535561 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.535530 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-podinfo\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.535806 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.535787 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-967m4\" (UniqueName: \"kubernetes.io/projected/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-kube-api-access-967m4\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.535861 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.535830 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-data\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.573442 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.573416 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b2a05c-openshift-default-dockercfg-lgjtt\"" Oct 30 01:23:45.637959 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.637917 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-credential-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.638156 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.638074 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-envoy\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.638156 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.638113 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istiod-ca-cert\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.638277 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.638174 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-workload-certs\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.638277 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.638229 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-token\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.638277 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.638260 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-workload-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.638456 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.638286 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-podinfo\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.638516 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.638493 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-967m4\" (UniqueName: \"kubernetes.io/projected/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-kube-api-access-967m4\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.638574 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.638530 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-data\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.638639 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.638604 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-workload-certs\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.638699 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.638639 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-workload-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.638924 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.638896 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-credential-socket\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.639063 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.638929 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-data\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.639063 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.639039 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istiod-ca-cert\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.640598 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.640573 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-envoy\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.641149 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.641125 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-podinfo\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.659448 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.659423 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-token\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.660102 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.660081 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-967m4\" (UniqueName: \"kubernetes.io/projected/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-kube-api-access-967m4\") pod \"e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:45.670972 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.670950 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:45.786444 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:45.786414 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:46.023173 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.023139 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj"] Oct 30 01:23:46.027870 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.027849 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" Oct 30 01:23:46.050741 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.050686 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 01:23:46.050741 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.050686 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 01:23:46.050953 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.050901 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 01:23:46.051011 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.050960 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-txv2f\"" Oct 30 01:23:46.066986 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.066956 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj"] Oct 30 01:23:46.134033 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:23:46.134002 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d4bbf4_260b_4a7e_bf75_a16c37b7e526.slice/crio-226e5d590ac90b3d8e3ad53c4f6c5520d401e336ce95a450439e2224796aa5cf WatchSource:0}: Error finding container 226e5d590ac90b3d8e3ad53c4f6c5520d401e336ce95a450439e2224796aa5cf: Status 404 returned error can't find the container with id 226e5d590ac90b3d8e3ad53c4f6c5520d401e336ce95a450439e2224796aa5cf Oct 30 01:23:46.136011 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.135983 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666"] Oct 30 01:23:46.137194 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.137160 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:23:46.137282 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.137230 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:23:46.137282 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.137260 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:23:46.147814 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.147756 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8404a58-57e8-4ade-addc-c7b180433b35-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ppsvj\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" Oct 30 01:23:46.147920 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.147888 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8404a58-57e8-4ade-addc-c7b180433b35-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ppsvj\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" Oct 30 01:23:46.147985 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.147929 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxh4k\" (UniqueName: \"kubernetes.io/projected/c8404a58-57e8-4ade-addc-c7b180433b35-kube-api-access-gxh4k\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ppsvj\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" Oct 30 01:23:46.148037 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.148019 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8404a58-57e8-4ade-addc-c7b180433b35-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ppsvj\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" Oct 30 01:23:46.148151 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.148127 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8404a58-57e8-4ade-addc-c7b180433b35-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ppsvj\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" Oct 30 01:23:46.148335 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.148312 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8404a58-57e8-4ade-addc-c7b180433b35-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ppsvj\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" Oct 30 01:23:46.148428 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.148412 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8404a58-57e8-4ade-addc-c7b180433b35-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ppsvj\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" Oct 30 01:23:46.250076 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.250042 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8404a58-57e8-4ade-addc-c7b180433b35-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ppsvj\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" Oct 30 01:23:46.250239 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.250198 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8404a58-57e8-4ade-addc-c7b180433b35-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ppsvj\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" Oct 30 01:23:46.250328 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.250256 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8404a58-57e8-4ade-addc-c7b180433b35-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ppsvj\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" Oct 30 01:23:46.250328 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.250293 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxh4k\" (UniqueName: \"kubernetes.io/projected/c8404a58-57e8-4ade-addc-c7b180433b35-kube-api-access-gxh4k\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ppsvj\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" Oct 30 01:23:46.250439 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.250361 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8404a58-57e8-4ade-addc-c7b180433b35-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ppsvj\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" Oct 30 01:23:46.250486 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.250470 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8404a58-57e8-4ade-addc-c7b180433b35-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ppsvj\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" Oct 30 01:23:46.250645 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.250621 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8404a58-57e8-4ade-addc-c7b180433b35-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ppsvj\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" Oct 30 01:23:46.251482 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.251442 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8404a58-57e8-4ade-addc-c7b180433b35-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ppsvj\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" Oct 30 01:23:46.253151 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.253116 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8404a58-57e8-4ade-addc-c7b180433b35-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ppsvj\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" Oct 30 01:23:46.253263 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.253132 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8404a58-57e8-4ade-addc-c7b180433b35-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ppsvj\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" Oct 30 01:23:46.253408 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.253392 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8404a58-57e8-4ade-addc-c7b180433b35-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ppsvj\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" Oct 30 01:23:46.253451 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.253402 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8404a58-57e8-4ade-addc-c7b180433b35-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ppsvj\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" Oct 30 01:23:46.273173 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.273149 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8404a58-57e8-4ade-addc-c7b180433b35-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ppsvj\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" Oct 30 01:23:46.273834 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.273814 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxh4k\" (UniqueName: \"kubernetes.io/projected/c8404a58-57e8-4ade-addc-c7b180433b35-kube-api-access-gxh4k\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ppsvj\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" Oct 30 01:23:46.342938 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.342853 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" Oct 30 01:23:46.707522 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.706910 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj"] Oct 30 01:23:46.712578 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:23:46.712543 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8404a58_57e8_4ade_addc_c7b180433b35.slice/crio-532079d2e012a0b45133bdb527a19fb9cda6686a75092a10550ad95075fe95f1 WatchSource:0}: Error finding container 532079d2e012a0b45133bdb527a19fb9cda6686a75092a10550ad95075fe95f1: Status 404 returned error can't find the container with id 532079d2e012a0b45133bdb527a19fb9cda6686a75092a10550ad95075fe95f1 Oct 30 01:23:46.716753 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.716720 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:23:46.716898 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.716813 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:23:46.834175 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.834138 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" event={"ID":"94d4bbf4-260b-4a7e-bf75-a16c37b7e526","Type":"ContainerStarted","Data":"b2420f140b7665153a28c200bb2423adad3c92534cae5abaea824eddcb4d16b6"} Oct 30 01:23:46.834344 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.834203 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" event={"ID":"94d4bbf4-260b-4a7e-bf75-a16c37b7e526","Type":"ContainerStarted","Data":"226e5d590ac90b3d8e3ad53c4f6c5520d401e336ce95a450439e2224796aa5cf"} Oct 30 01:23:46.836033 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.836006 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" event={"ID":"c8404a58-57e8-4ade-addc-c7b180433b35","Type":"ContainerStarted","Data":"39fe6b535c6fbb35041bd132ddf410a95abb5ec5760ee9345bc23ffd4607d546"} Oct 30 01:23:46.836147 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.836049 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" event={"ID":"c8404a58-57e8-4ade-addc-c7b180433b35","Type":"ContainerStarted","Data":"532079d2e012a0b45133bdb527a19fb9cda6686a75092a10550ad95075fe95f1"} Oct 30 01:23:46.836147 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.836073 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" Oct 30 01:23:46.890447 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.890375 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" podStartSLOduration=1.890355789 podStartE2EDuration="1.890355789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:46.88009624 +0000 UTC m=+2427.195964566" watchObservedRunningTime="2026-10-30 01:23:46.890355789 +0000 UTC m=+2427.206224123" Oct 30 01:23:46.927994 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:46.927937 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" podStartSLOduration=1.927922198 podStartE2EDuration="1.927922198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:46.925883836 +0000 UTC m=+2427.241752160" watchObservedRunningTime="2026-10-30 01:23:46.927922198 +0000 UTC m=+2427.243790525" Oct 30 01:23:47.614500 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:47.614459 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:47.799785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:47.795125 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:47.824783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:47.821008 5300 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 01:23:47.824783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:47.821068 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" podUID="94d4bbf4-260b-4a7e-bf75-a16c37b7e526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 01:23:47.913575 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:47.913393 5300 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-ppsvj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:23:47.913575 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:47.913446 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" podUID="c8404a58-57e8-4ade-addc-c7b180433b35" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:23:48.787776 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:48.787724 5300 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 01:23:48.787982 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:48.787802 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" podUID="94d4bbf4-260b-4a7e-bf75-a16c37b7e526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 01:23:49.682681 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:49.682648 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:49.787533 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:49.787498 5300 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 01:23:49.787718 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:49.787552 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" podUID="94d4bbf4-260b-4a7e-bf75-a16c37b7e526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 01:23:50.306533 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:50.305644 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" Oct 30 01:23:50.348991 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:50.348947 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" Oct 30 01:23:50.786793 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:50.786725 5300 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 01:23:50.787292 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:50.786799 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" podUID="94d4bbf4-260b-4a7e-bf75-a16c37b7e526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 01:23:50.910117 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:50.910085 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" Oct 30 01:23:50.968268 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:50.968235 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" Oct 30 01:23:51.569201 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:51.569167 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:51.596561 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:23:51.596530 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:23:51.787311 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:51.787282 5300 patch_prober.go:29] interesting pod/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 01:23:51.787740 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:51.787338 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" podUID="94d4bbf4-260b-4a7e-bf75-a16c37b7e526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 01:23:52.793224 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:52.793191 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:52.793721 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:52.793699 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:52.794616 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:52.794595 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:23:53.126991 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:53.126903 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:54.759286 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:54.759245 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:56.283207 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:56.283173 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps"] Oct 30 01:23:56.318597 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:56.318548 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:56.430843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:56.426476 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:57.883026 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:57.882956 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:58.605189 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:58.605152 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts"] Oct 30 01:23:58.615921 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:58.615890 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:59.331702 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:59.331666 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:59.784164 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:59.784132 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:23:59.883239 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:23:59.883205 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:00.992643 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:00.992551 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:01.245508 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:01.245412 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:02.801006 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:02.800961 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:04.071118 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:04.071070 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:04.180699 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:04.180668 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:05.202821 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:05.202776 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:05.311042 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:05.311011 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:05.634247 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:05.634210 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:06.174466 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:06.174432 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:06.280576 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:06.280541 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:06.348597 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:06.348559 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd"] Oct 30 01:24:06.355373 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:06.355350 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd" Oct 30 01:24:06.370933 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:06.370903 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd"] Oct 30 01:24:06.379297 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:06.379268 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a6b650-dockercfg-4z8zw\"" Oct 30 01:24:06.379477 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:06.379333 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a6b650\"" Oct 30 01:24:06.673835 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:06.673723 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsbdb\" (UniqueName: \"kubernetes.io/projected/5ba6e8c3-2254-495b-9f97-55fba6c120c8-kube-api-access-hsbdb\") pod \"payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd\" (UID: \"5ba6e8c3-2254-495b-9f97-55fba6c120c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd" Oct 30 01:24:06.684078 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:06.684048 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ba6e8c3-2254-495b-9f97-55fba6c120c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd\" (UID: \"5ba6e8c3-2254-495b-9f97-55fba6c120c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd" Oct 30 01:24:06.786740 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:06.786699 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsbdb\" (UniqueName: \"kubernetes.io/projected/5ba6e8c3-2254-495b-9f97-55fba6c120c8-kube-api-access-hsbdb\") pod \"payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd\" (UID: \"5ba6e8c3-2254-495b-9f97-55fba6c120c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd" Oct 30 01:24:06.786943 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:06.786748 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ba6e8c3-2254-495b-9f97-55fba6c120c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd\" (UID: \"5ba6e8c3-2254-495b-9f97-55fba6c120c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd" Oct 30 01:24:06.787572 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:06.787543 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ba6e8c3-2254-495b-9f97-55fba6c120c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd\" (UID: \"5ba6e8c3-2254-495b-9f97-55fba6c120c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd" Oct 30 01:24:06.807596 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:06.807566 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsbdb\" (UniqueName: \"kubernetes.io/projected/5ba6e8c3-2254-495b-9f97-55fba6c120c8-kube-api-access-hsbdb\") pod \"payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd\" (UID: \"5ba6e8c3-2254-495b-9f97-55fba6c120c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd" Oct 30 01:24:06.981414 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:06.981383 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd" Oct 30 01:24:07.164223 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:07.164185 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:07.307050 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:07.306981 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd"] Oct 30 01:24:07.312965 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:24:07.312911 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba6e8c3_2254_495b_9f97_55fba6c120c8.slice/crio-e5d8722b509b75c647b1d7cb31dd4f513f1b895174870b97d1ade7642b15ab66 WatchSource:0}: Error finding container e5d8722b509b75c647b1d7cb31dd4f513f1b895174870b97d1ade7642b15ab66: Status 404 returned error can't find the container with id e5d8722b509b75c647b1d7cb31dd4f513f1b895174870b97d1ade7642b15ab66 Oct 30 01:24:08.114304 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:08.114262 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd" event={"ID":"5ba6e8c3-2254-495b-9f97-55fba6c120c8","Type":"ContainerStarted","Data":"e77f3db2491f6508515e835df4899884cc275247b7f0c99f50d1571682e7a2aa"} Oct 30 01:24:08.114503 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:08.114340 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd" Oct 30 01:24:08.114503 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:08.114363 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd" event={"ID":"5ba6e8c3-2254-495b-9f97-55fba6c120c8","Type":"ContainerStarted","Data":"e5d8722b509b75c647b1d7cb31dd4f513f1b895174870b97d1ade7642b15ab66"} Oct 30 01:24:08.156928 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:08.156881 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd" podStartSLOduration=2.156868464 podStartE2EDuration="2.156868464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:08.155044952 +0000 UTC m=+2448.470913276" watchObservedRunningTime="2026-10-30 01:24:08.156868464 +0000 UTC m=+2448.472736789" Oct 30 01:24:08.791688 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:08.791658 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:10.332715 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:10.332669 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:11.309464 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:11.309263 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" podUID="0957caa6-ff6b-4cf8-be88-717b6434a22f" containerName="main" containerID="cri-o://d6c750959fc74b4fa37f00b4fde3105f8673d49cfef299b43c65b67810f134e2" gracePeriod=45 Oct 30 01:24:11.608269 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:11.608244 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" Oct 30 01:24:11.665077 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:11.665043 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0957caa6-ff6b-4cf8-be88-717b6434a22f-kube-api-access-hljx2\" (UniqueName: \"kubernetes.io/projected/0957caa6-ff6b-4cf8-be88-717b6434a22f-kube-api-access-hljx2\") pod \"0957caa6-ff6b-4cf8-be88-717b6434a22f\" (UID: \"0957caa6-ff6b-4cf8-be88-717b6434a22f\") " Oct 30 01:24:11.665225 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:11.665104 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-model-cache\") pod \"0957caa6-ff6b-4cf8-be88-717b6434a22f\" (UID: \"0957caa6-ff6b-4cf8-be88-717b6434a22f\") " Oct 30 01:24:11.665225 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:11.665200 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-tmp-dir\") pod \"0957caa6-ff6b-4cf8-be88-717b6434a22f\" (UID: \"0957caa6-ff6b-4cf8-be88-717b6434a22f\") " Oct 30 01:24:11.665305 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:11.665292 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-home\" (UniqueName: \"kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-home\") pod \"0957caa6-ff6b-4cf8-be88-717b6434a22f\" (UID: \"0957caa6-ff6b-4cf8-be88-717b6434a22f\") " Oct 30 01:24:11.665370 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:11.665355 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0957caa6-ff6b-4cf8-be88-717b6434a22f-tls-certs\" (UniqueName: \"kubernetes.io/secret/0957caa6-ff6b-4cf8-be88-717b6434a22f-tls-certs\") pod \"0957caa6-ff6b-4cf8-be88-717b6434a22f\" (UID: \"0957caa6-ff6b-4cf8-be88-717b6434a22f\") " Oct 30 01:24:11.665546 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:11.665530 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-dshm\") pod \"0957caa6-ff6b-4cf8-be88-717b6434a22f\" (UID: \"0957caa6-ff6b-4cf8-be88-717b6434a22f\") " Oct 30 01:24:11.666918 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:11.666845 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-tmp-dir" pod "0957caa6-ff6b-4cf8-be88-717b6434a22f" (UID: "0957caa6-ff6b-4cf8-be88-717b6434a22f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:11.667058 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:11.667015 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-model-cache" pod "0957caa6-ff6b-4cf8-be88-717b6434a22f" (UID: "0957caa6-ff6b-4cf8-be88-717b6434a22f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:11.667190 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:11.667159 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-home" pod "0957caa6-ff6b-4cf8-be88-717b6434a22f" (UID: "0957caa6-ff6b-4cf8-be88-717b6434a22f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:11.669246 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:11.669160 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-dshm" pod "0957caa6-ff6b-4cf8-be88-717b6434a22f" (UID: "0957caa6-ff6b-4cf8-be88-717b6434a22f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:11.669246 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:11.669225 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0957caa6-ff6b-4cf8-be88-717b6434a22f-kube-api-access-hljx2" pod "0957caa6-ff6b-4cf8-be88-717b6434a22f" (UID: "0957caa6-ff6b-4cf8-be88-717b6434a22f"). InnerVolumeSpecName "kube-api-access-hljx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:11.669378 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:11.669241 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0957caa6-ff6b-4cf8-be88-717b6434a22f-tls-certs" pod "0957caa6-ff6b-4cf8-be88-717b6434a22f" (UID: "0957caa6-ff6b-4cf8-be88-717b6434a22f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:11.767915 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:11.767882 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hljx2\" (UniqueName: \"kubernetes.io/projected/0957caa6-ff6b-4cf8-be88-717b6434a22f-kube-api-access-hljx2\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.767915 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:11.767911 5300 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-model-cache\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.768091 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:11.767927 5300 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-tmp-dir\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.768091 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:11.767938 5300 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-home\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.768091 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:11.767948 5300 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0957caa6-ff6b-4cf8-be88-717b6434a22f-tls-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.768091 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:11.767958 5300 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0957caa6-ff6b-4cf8-be88-717b6434a22f-dshm\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:24:11.817834 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:11.817804 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:12.148503 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:12.148410 5300 generic.go:356] "Generic (PLEG): container finished" podID="0957caa6-ff6b-4cf8-be88-717b6434a22f" containerID="d6c750959fc74b4fa37f00b4fde3105f8673d49cfef299b43c65b67810f134e2" exitCode=0 Oct 30 01:24:12.148503 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:12.148454 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" Oct 30 01:24:12.148503 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:12.148491 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" event={"ID":"0957caa6-ff6b-4cf8-be88-717b6434a22f","Type":"ContainerDied","Data":"d6c750959fc74b4fa37f00b4fde3105f8673d49cfef299b43c65b67810f134e2"} Oct 30 01:24:12.148753 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:12.148537 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps" event={"ID":"0957caa6-ff6b-4cf8-be88-717b6434a22f","Type":"ContainerDied","Data":"f53f1cc757035a227dc1b2473cd4d248bc4bd844db45f3b04def8872a162842a"} Oct 30 01:24:12.148753 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:12.148555 5300 scope.go:122] "RemoveContainer" containerID="d6c750959fc74b4fa37f00b4fde3105f8673d49cfef299b43c65b67810f134e2" Oct 30 01:24:12.159126 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:12.159103 5300 scope.go:122] "RemoveContainer" containerID="d6c750959fc74b4fa37f00b4fde3105f8673d49cfef299b43c65b67810f134e2" Oct 30 01:24:12.159383 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:24:12.159364 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6c750959fc74b4fa37f00b4fde3105f8673d49cfef299b43c65b67810f134e2\": container with ID starting with d6c750959fc74b4fa37f00b4fde3105f8673d49cfef299b43c65b67810f134e2 not found: ID does not exist" containerID="d6c750959fc74b4fa37f00b4fde3105f8673d49cfef299b43c65b67810f134e2" Oct 30 01:24:12.159439 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:12.159391 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6c750959fc74b4fa37f00b4fde3105f8673d49cfef299b43c65b67810f134e2"} err="failed to get container status \"d6c750959fc74b4fa37f00b4fde3105f8673d49cfef299b43c65b67810f134e2\": rpc error: code = NotFound desc = could not find container \"d6c750959fc74b4fa37f00b4fde3105f8673d49cfef299b43c65b67810f134e2\": container with ID starting with d6c750959fc74b4fa37f00b4fde3105f8673d49cfef299b43c65b67810f134e2 not found: ID does not exist" Oct 30 01:24:12.206012 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:12.205988 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps"] Oct 30 01:24:12.220163 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:12.220135 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e2d6d3/auth-test-model-e2d6d3-kserve-7fd46d6785-6vtps"] Oct 30 01:24:13.240372 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:13.240343 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:13.336571 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:13.336537 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:13.631589 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:13.631492 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" podUID="bb602fa6-731e-4991-9504-7c743abb828b" containerName="main" containerID="cri-o://f0052e01936627d657d31b054e2ee386a695f5f1293e3011e6b99fa52acfbf2f" gracePeriod=45 Oct 30 01:24:13.832139 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:13.832105 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0957caa6-ff6b-4cf8-be88-717b6434a22f" path="/var/lib/kubelet/pods/0957caa6-ff6b-4cf8-be88-717b6434a22f/volumes" Oct 30 01:24:13.942257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:13.942221 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" Oct 30 01:24:14.004428 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.004399 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-home\" (UniqueName: \"kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-home\") pod \"bb602fa6-731e-4991-9504-7c743abb828b\" (UID: \"bb602fa6-731e-4991-9504-7c743abb828b\") " Oct 30 01:24:14.004583 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.004506 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-dshm\") pod \"bb602fa6-731e-4991-9504-7c743abb828b\" (UID: \"bb602fa6-731e-4991-9504-7c743abb828b\") " Oct 30 01:24:14.004583 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.004543 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-tmp-dir\") pod \"bb602fa6-731e-4991-9504-7c743abb828b\" (UID: \"bb602fa6-731e-4991-9504-7c743abb828b\") " Oct 30 01:24:14.004710 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.004608 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb602fa6-731e-4991-9504-7c743abb828b-tls-certs\" (UniqueName: \"kubernetes.io/secret/bb602fa6-731e-4991-9504-7c743abb828b-tls-certs\") pod \"bb602fa6-731e-4991-9504-7c743abb828b\" (UID: \"bb602fa6-731e-4991-9504-7c743abb828b\") " Oct 30 01:24:14.004784 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.004714 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-home" pod "bb602fa6-731e-4991-9504-7c743abb828b" (UID: "bb602fa6-731e-4991-9504-7c743abb828b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:14.004784 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.004745 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-model-cache\") pod \"bb602fa6-731e-4991-9504-7c743abb828b\" (UID: \"bb602fa6-731e-4991-9504-7c743abb828b\") " Oct 30 01:24:14.004990 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.004969 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb602fa6-731e-4991-9504-7c743abb828b-kube-api-access-bqgr4\" (UniqueName: \"kubernetes.io/projected/bb602fa6-731e-4991-9504-7c743abb828b-kube-api-access-bqgr4\") pod \"bb602fa6-731e-4991-9504-7c743abb828b\" (UID: \"bb602fa6-731e-4991-9504-7c743abb828b\") " Oct 30 01:24:14.005389 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.005084 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-model-cache" pod "bb602fa6-731e-4991-9504-7c743abb828b" (UID: "bb602fa6-731e-4991-9504-7c743abb828b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:14.005389 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.005135 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-tmp-dir" pod "bb602fa6-731e-4991-9504-7c743abb828b" (UID: "bb602fa6-731e-4991-9504-7c743abb828b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:14.005787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.005740 5300 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-model-cache\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:24:14.005787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.005782 5300 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-home\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:24:14.005947 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.005817 5300 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-tmp-dir\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:24:14.014652 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.013321 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb602fa6-731e-4991-9504-7c743abb828b-tls-certs" pod "bb602fa6-731e-4991-9504-7c743abb828b" (UID: "bb602fa6-731e-4991-9504-7c743abb828b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:14.017642 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.017609 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb602fa6-731e-4991-9504-7c743abb828b-kube-api-access-bqgr4" pod "bb602fa6-731e-4991-9504-7c743abb828b" (UID: "bb602fa6-731e-4991-9504-7c743abb828b"). InnerVolumeSpecName "kube-api-access-bqgr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:14.022719 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.021347 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-dshm" pod "bb602fa6-731e-4991-9504-7c743abb828b" (UID: "bb602fa6-731e-4991-9504-7c743abb828b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:14.107287 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.107252 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqgr4\" (UniqueName: \"kubernetes.io/projected/bb602fa6-731e-4991-9504-7c743abb828b-kube-api-access-bqgr4\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:24:14.107287 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.107281 5300 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb602fa6-731e-4991-9504-7c743abb828b-dshm\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:24:14.107287 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.107292 5300 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb602fa6-731e-4991-9504-7c743abb828b-tls-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:24:14.167289 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.167201 5300 generic.go:356] "Generic (PLEG): container finished" podID="bb602fa6-731e-4991-9504-7c743abb828b" containerID="f0052e01936627d657d31b054e2ee386a695f5f1293e3011e6b99fa52acfbf2f" exitCode=0 Oct 30 01:24:14.167289 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.167250 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" Oct 30 01:24:14.167289 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.167262 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" event={"ID":"bb602fa6-731e-4991-9504-7c743abb828b","Type":"ContainerDied","Data":"f0052e01936627d657d31b054e2ee386a695f5f1293e3011e6b99fa52acfbf2f"} Oct 30 01:24:14.167556 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.167300 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts" event={"ID":"bb602fa6-731e-4991-9504-7c743abb828b","Type":"ContainerDied","Data":"1e5c606bc84bba5a60866004336f7d4b929634449aef38ca8044e865a363051f"} Oct 30 01:24:14.167556 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.167323 5300 scope.go:122] "RemoveContainer" containerID="f0052e01936627d657d31b054e2ee386a695f5f1293e3011e6b99fa52acfbf2f" Oct 30 01:24:14.178636 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.178620 5300 scope.go:122] "RemoveContainer" containerID="f0052e01936627d657d31b054e2ee386a695f5f1293e3011e6b99fa52acfbf2f" Oct 30 01:24:14.178905 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:24:14.178888 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0052e01936627d657d31b054e2ee386a695f5f1293e3011e6b99fa52acfbf2f\": container with ID starting with f0052e01936627d657d31b054e2ee386a695f5f1293e3011e6b99fa52acfbf2f not found: ID does not exist" containerID="f0052e01936627d657d31b054e2ee386a695f5f1293e3011e6b99fa52acfbf2f" Oct 30 01:24:14.178972 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.178912 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0052e01936627d657d31b054e2ee386a695f5f1293e3011e6b99fa52acfbf2f"} err="failed to get container status \"f0052e01936627d657d31b054e2ee386a695f5f1293e3011e6b99fa52acfbf2f\": rpc error: code = NotFound desc = could not find container \"f0052e01936627d657d31b054e2ee386a695f5f1293e3011e6b99fa52acfbf2f\": container with ID starting with f0052e01936627d657d31b054e2ee386a695f5f1293e3011e6b99fa52acfbf2f not found: ID does not exist" Oct 30 01:24:14.225277 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.225252 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts"] Oct 30 01:24:14.230627 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.230604 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-999ff9/auth-test-model-999ff9-kserve-6958bb4d77-5rvts"] Oct 30 01:24:14.988952 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:14.988904 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:15.825988 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:15.825954 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb602fa6-731e-4991-9504-7c743abb828b" path="/var/lib/kubelet/pods/bb602fa6-731e-4991-9504-7c743abb828b/volumes" Oct 30 01:24:16.442560 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:16.442521 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:18.591221 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:18.591176 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:19.125521 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:19.125492 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd" Oct 30 01:24:19.626116 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:19.626083 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:19.768800 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:19.768728 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:20.589268 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:20.589231 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2"] Oct 30 01:24:20.640568 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:20.640532 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:20.769411 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:20.769377 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:21.594705 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:24:21.594668 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:24:22.327661 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:22.327621 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:23.749174 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:23.749137 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:25.548098 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:25.548061 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:28.007499 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:28.007438 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:28.139916 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:28.139880 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx"] Oct 30 01:24:28.155328 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:28.155248 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:28.785787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:28.785223 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:29.870534 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:29.870485 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:29.983326 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:29.983288 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:30.708308 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:30.708274 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:31.542514 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:31.542471 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:31.699136 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:31.699096 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:32.424046 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:32.424015 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:32.545852 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:32.545807 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:33.381168 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:33.381138 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:34.606084 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:34.606042 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:34.795061 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:34.795027 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:35.622977 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:35.622935 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" podUID="ff735ea9-e315-46b8-b613-8164e6a2823a" containerName="main" containerID="cri-o://bbed55714c76a1608f6259095b6d39a19dec6628797f808311c821acc2d00e37" gracePeriod=45 Oct 30 01:24:35.914942 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:35.914916 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" Oct 30 01:24:36.017636 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.017602 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff735ea9-e315-46b8-b613-8164e6a2823a-tls-certs\" (UniqueName: \"kubernetes.io/secret/ff735ea9-e315-46b8-b613-8164e6a2823a-tls-certs\") pod \"ff735ea9-e315-46b8-b613-8164e6a2823a\" (UID: \"ff735ea9-e315-46b8-b613-8164e6a2823a\") " Oct 30 01:24:36.017846 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.017723 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-tmp-dir\") pod \"ff735ea9-e315-46b8-b613-8164e6a2823a\" (UID: \"ff735ea9-e315-46b8-b613-8164e6a2823a\") " Oct 30 01:24:36.017994 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.017976 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-model-cache\") pod \"ff735ea9-e315-46b8-b613-8164e6a2823a\" (UID: \"ff735ea9-e315-46b8-b613-8164e6a2823a\") " Oct 30 01:24:36.018058 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.018030 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-dshm\") pod \"ff735ea9-e315-46b8-b613-8164e6a2823a\" (UID: \"ff735ea9-e315-46b8-b613-8164e6a2823a\") " Oct 30 01:24:36.018116 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.018085 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-home\" (UniqueName: \"kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-home\") pod \"ff735ea9-e315-46b8-b613-8164e6a2823a\" (UID: \"ff735ea9-e315-46b8-b613-8164e6a2823a\") " Oct 30 01:24:36.018116 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.018083 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-tmp-dir" pod "ff735ea9-e315-46b8-b613-8164e6a2823a" (UID: "ff735ea9-e315-46b8-b613-8164e6a2823a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:36.018231 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.018156 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff735ea9-e315-46b8-b613-8164e6a2823a-kube-api-access-xpl7m\" (UniqueName: \"kubernetes.io/projected/ff735ea9-e315-46b8-b613-8164e6a2823a-kube-api-access-xpl7m\") pod \"ff735ea9-e315-46b8-b613-8164e6a2823a\" (UID: \"ff735ea9-e315-46b8-b613-8164e6a2823a\") " Oct 30 01:24:36.018320 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.018295 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-model-cache" pod "ff735ea9-e315-46b8-b613-8164e6a2823a" (UID: "ff735ea9-e315-46b8-b613-8164e6a2823a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:36.018816 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.018497 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-home" pod "ff735ea9-e315-46b8-b613-8164e6a2823a" (UID: "ff735ea9-e315-46b8-b613-8164e6a2823a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:36.019255 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.019185 5300 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-home\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:24:36.019255 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.019211 5300 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-tmp-dir\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:24:36.019255 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.019228 5300 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-model-cache\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:24:36.020959 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.020935 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff735ea9-e315-46b8-b613-8164e6a2823a-tls-certs" pod "ff735ea9-e315-46b8-b613-8164e6a2823a" (UID: "ff735ea9-e315-46b8-b613-8164e6a2823a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:36.020959 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.020949 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-dshm" pod "ff735ea9-e315-46b8-b613-8164e6a2823a" (UID: "ff735ea9-e315-46b8-b613-8164e6a2823a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:36.021390 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.021368 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff735ea9-e315-46b8-b613-8164e6a2823a-kube-api-access-xpl7m" pod "ff735ea9-e315-46b8-b613-8164e6a2823a" (UID: "ff735ea9-e315-46b8-b613-8164e6a2823a"). InnerVolumeSpecName "kube-api-access-xpl7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:36.120882 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.120847 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpl7m\" (UniqueName: \"kubernetes.io/projected/ff735ea9-e315-46b8-b613-8164e6a2823a-kube-api-access-xpl7m\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:24:36.120882 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.120877 5300 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff735ea9-e315-46b8-b613-8164e6a2823a-tls-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:24:36.120882 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.120887 5300 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff735ea9-e315-46b8-b613-8164e6a2823a-dshm\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:24:36.259890 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.259858 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:36.457582 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.457548 5300 generic.go:356] "Generic (PLEG): container finished" podID="ff735ea9-e315-46b8-b613-8164e6a2823a" containerID="bbed55714c76a1608f6259095b6d39a19dec6628797f808311c821acc2d00e37" exitCode=0 Oct 30 01:24:36.457772 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.457588 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" Oct 30 01:24:36.457772 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.457628 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" event={"ID":"ff735ea9-e315-46b8-b613-8164e6a2823a","Type":"ContainerDied","Data":"bbed55714c76a1608f6259095b6d39a19dec6628797f808311c821acc2d00e37"} Oct 30 01:24:36.457772 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.457675 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2" event={"ID":"ff735ea9-e315-46b8-b613-8164e6a2823a","Type":"ContainerDied","Data":"908222fb45fb16e83a3d5d6e9b6e2a660364e9f6d80ff849f1103b56fff769ed"} Oct 30 01:24:36.457772 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.457693 5300 scope.go:122] "RemoveContainer" containerID="bbed55714c76a1608f6259095b6d39a19dec6628797f808311c821acc2d00e37" Oct 30 01:24:36.470671 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.470654 5300 scope.go:122] "RemoveContainer" containerID="bbed55714c76a1608f6259095b6d39a19dec6628797f808311c821acc2d00e37" Oct 30 01:24:36.470976 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:24:36.470956 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbed55714c76a1608f6259095b6d39a19dec6628797f808311c821acc2d00e37\": container with ID starting with bbed55714c76a1608f6259095b6d39a19dec6628797f808311c821acc2d00e37 not found: ID does not exist" containerID="bbed55714c76a1608f6259095b6d39a19dec6628797f808311c821acc2d00e37" Oct 30 01:24:36.471060 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.470983 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbed55714c76a1608f6259095b6d39a19dec6628797f808311c821acc2d00e37"} err="failed to get container status \"bbed55714c76a1608f6259095b6d39a19dec6628797f808311c821acc2d00e37\": rpc error: code = NotFound desc = could not find container \"bbed55714c76a1608f6259095b6d39a19dec6628797f808311c821acc2d00e37\": container with ID starting with bbed55714c76a1608f6259095b6d39a19dec6628797f808311c821acc2d00e37 not found: ID does not exist" Oct 30 01:24:36.516914 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.516835 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2"] Oct 30 01:24:36.530769 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:36.530740 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e4f8a6/sub-test-model-e4f8a6-kserve-78c44749d5-cbqf2"] Oct 30 01:24:37.825691 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:37.825654 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff735ea9-e315-46b8-b613-8164e6a2823a" path="/var/lib/kubelet/pods/ff735ea9-e315-46b8-b613-8164e6a2823a/volumes" Oct 30 01:24:37.838900 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:37.838879 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:39.368638 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:39.368606 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:40.892600 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:40.892569 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:42.427952 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:42.427911 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:43.164991 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.164945 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" podUID="3f4fe743-0164-4a05-a608-5d7078948f3a" containerName="main" containerID="cri-o://b2a556e509bc043c32e850c50e09d3f69f4a794b6d74d593738e44e842463c12" gracePeriod=45 Oct 30 01:24:43.469492 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.469468 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" Oct 30 01:24:43.519674 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.519640 5300 generic.go:356] "Generic (PLEG): container finished" podID="3f4fe743-0164-4a05-a608-5d7078948f3a" containerID="b2a556e509bc043c32e850c50e09d3f69f4a794b6d74d593738e44e842463c12" exitCode=0 Oct 30 01:24:43.519874 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.519807 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" event={"ID":"3f4fe743-0164-4a05-a608-5d7078948f3a","Type":"ContainerDied","Data":"b2a556e509bc043c32e850c50e09d3f69f4a794b6d74d593738e44e842463c12"} Oct 30 01:24:43.519874 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.519844 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" event={"ID":"3f4fe743-0164-4a05-a608-5d7078948f3a","Type":"ContainerDied","Data":"34f8b4355f576e6fa6a8739a31d9b329fbc53891d1510257c4fcb3dc7545a080"} Oct 30 01:24:43.519874 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.519867 5300 scope.go:122] "RemoveContainer" containerID="b2a556e509bc043c32e850c50e09d3f69f4a794b6d74d593738e44e842463c12" Oct 30 01:24:43.520060 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.520006 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" Oct 30 01:24:43.535168 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.535138 5300 scope.go:122] "RemoveContainer" containerID="b2a556e509bc043c32e850c50e09d3f69f4a794b6d74d593738e44e842463c12" Oct 30 01:24:43.535733 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:24:43.535708 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2a556e509bc043c32e850c50e09d3f69f4a794b6d74d593738e44e842463c12\": container with ID starting with b2a556e509bc043c32e850c50e09d3f69f4a794b6d74d593738e44e842463c12 not found: ID does not exist" containerID="b2a556e509bc043c32e850c50e09d3f69f4a794b6d74d593738e44e842463c12" Oct 30 01:24:43.535881 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.535739 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2a556e509bc043c32e850c50e09d3f69f4a794b6d74d593738e44e842463c12"} err="failed to get container status \"b2a556e509bc043c32e850c50e09d3f69f4a794b6d74d593738e44e842463c12\": rpc error: code = NotFound desc = could not find container \"b2a556e509bc043c32e850c50e09d3f69f4a794b6d74d593738e44e842463c12\": container with ID starting with b2a556e509bc043c32e850c50e09d3f69f4a794b6d74d593738e44e842463c12 not found: ID does not exist" Oct 30 01:24:43.568007 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.567975 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f4fe743-0164-4a05-a608-5d7078948f3a-kube-api-access-m4pbz\" (UniqueName: \"kubernetes.io/projected/3f4fe743-0164-4a05-a608-5d7078948f3a-kube-api-access-m4pbz\") pod \"3f4fe743-0164-4a05-a608-5d7078948f3a\" (UID: \"3f4fe743-0164-4a05-a608-5d7078948f3a\") " Oct 30 01:24:43.568155 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.568021 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-dshm\") pod \"3f4fe743-0164-4a05-a608-5d7078948f3a\" (UID: \"3f4fe743-0164-4a05-a608-5d7078948f3a\") " Oct 30 01:24:43.568254 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.568237 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-tmp-dir\") pod \"3f4fe743-0164-4a05-a608-5d7078948f3a\" (UID: \"3f4fe743-0164-4a05-a608-5d7078948f3a\") " Oct 30 01:24:43.568370 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.568352 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-model-cache\") pod \"3f4fe743-0164-4a05-a608-5d7078948f3a\" (UID: \"3f4fe743-0164-4a05-a608-5d7078948f3a\") " Oct 30 01:24:43.568441 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.568420 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f4fe743-0164-4a05-a608-5d7078948f3a-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f4fe743-0164-4a05-a608-5d7078948f3a-tls-certs\") pod \"3f4fe743-0164-4a05-a608-5d7078948f3a\" (UID: \"3f4fe743-0164-4a05-a608-5d7078948f3a\") " Oct 30 01:24:43.568506 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.568465 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-home\" (UniqueName: \"kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-home\") pod \"3f4fe743-0164-4a05-a608-5d7078948f3a\" (UID: \"3f4fe743-0164-4a05-a608-5d7078948f3a\") " Oct 30 01:24:43.568938 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.568893 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-model-cache" pod "3f4fe743-0164-4a05-a608-5d7078948f3a" (UID: "3f4fe743-0164-4a05-a608-5d7078948f3a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:43.568938 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.568917 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-home" pod "3f4fe743-0164-4a05-a608-5d7078948f3a" (UID: "3f4fe743-0164-4a05-a608-5d7078948f3a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:43.569332 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.569312 5300 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-model-cache\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:24:43.569418 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.569338 5300 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-home\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:24:43.569418 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.569398 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-tmp-dir" pod "3f4fe743-0164-4a05-a608-5d7078948f3a" (UID: "3f4fe743-0164-4a05-a608-5d7078948f3a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:43.570613 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.570586 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-dshm" pod "3f4fe743-0164-4a05-a608-5d7078948f3a" (UID: "3f4fe743-0164-4a05-a608-5d7078948f3a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:24:43.570883 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.570860 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4fe743-0164-4a05-a608-5d7078948f3a-kube-api-access-m4pbz" pod "3f4fe743-0164-4a05-a608-5d7078948f3a" (UID: "3f4fe743-0164-4a05-a608-5d7078948f3a"). InnerVolumeSpecName "kube-api-access-m4pbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:43.571240 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.571217 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f4fe743-0164-4a05-a608-5d7078948f3a-tls-certs" pod "3f4fe743-0164-4a05-a608-5d7078948f3a" (UID: "3f4fe743-0164-4a05-a608-5d7078948f3a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:43.671204 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.671170 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4pbz\" (UniqueName: \"kubernetes.io/projected/3f4fe743-0164-4a05-a608-5d7078948f3a-kube-api-access-m4pbz\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:24:43.671204 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.671205 5300 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-dshm\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:24:43.671401 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.671221 5300 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f4fe743-0164-4a05-a608-5d7078948f3a-tmp-dir\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:24:43.671401 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.671234 5300 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f4fe743-0164-4a05-a608-5d7078948f3a-tls-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:24:43.875239 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.875206 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx"] Oct 30 01:24:43.888932 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:43.888900 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx"] Oct 30 01:24:44.011984 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:44.011951 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:44.393924 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:44.393885 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-886adb/sub-test-model-886adb-kserve-844f56697f-zlgnx" podUID="3f4fe743-0164-4a05-a608-5d7078948f3a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.151:8000/ready\": context deadline exceeded" Oct 30 01:24:45.457012 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:45.456945 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:45.825003 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:45.824970 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f4fe743-0164-4a05-a608-5d7078948f3a" path="/var/lib/kubelet/pods/3f4fe743-0164-4a05-a608-5d7078948f3a/volumes" Oct 30 01:24:47.126040 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:47.126010 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:48.633593 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:48.633553 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:49.928039 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:49.928001 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:50.215174 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:50.215095 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:50.318139 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:50.318105 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:50.629498 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:50.629451 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:51.598364 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:24:51.598325 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:24:51.944286 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:51.944187 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:53.930969 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:53.930903 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:55.414150 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:55.414117 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:56.743075 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:56.743028 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:56.842437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:56.842409 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:57.025172 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:57.025048 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:58.453028 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:58.452988 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:24:59.884479 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:24:59.884442 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:01.521596 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:01.521560 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:02.923435 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:02.923403 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:03.818824 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:03.818791 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-nkqms_7817f490-6fc5-4d17-8acc-f83273bd9a5c/payload-processing/0.log" Oct 30 01:25:03.993707 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:03.993681 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-nkqms_7817f490-6fc5-4d17-8acc-f83273bd9a5c/payload-processing/0.log" Oct 30 01:25:04.363194 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:04.363157 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:06.674191 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:06.674162 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:08.735128 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:08.735097 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:10.300704 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:10.300664 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:11.827306 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:11.827274 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:13.309995 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:13.309957 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:14.508683 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:14.508641 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:14.861833 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:14.860000 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:15.652704 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:15.652667 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:17.330267 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:17.330233 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:18.845670 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:18.845638 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:20.414454 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:20.414393 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:21.596142 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:25:21.596106 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:25:21.999851 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:21.999818 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:23.412776 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:23.412732 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:25.120240 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:25.120198 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:26.638144 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:26.638108 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:28.147218 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:28.147170 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:29.709777 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:29.709731 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:30.923339 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:30.923259 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq"] Oct 30 01:25:30.923843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:30.923479 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" podUID="2a68439c-dee6-4fb4-857d-bd09bbe8e214" containerName="istio-proxy" containerID="cri-o://0cd3d38e51e39bd6f8f0fba36c772bb4f64f3d5a09566d8b8a5fe80b821c0cd4" gracePeriod=30 Oct 30 01:25:31.311094 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:31.311041 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:32.734434 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:32.734395 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:34.422713 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:34.422672 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:34.810126 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:34.810084 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:35.947610 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:35.947583 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:36.100602 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.100571 5300 generic.go:356] "Generic (PLEG): container finished" podID="2a68439c-dee6-4fb4-857d-bd09bbe8e214" containerID="0cd3d38e51e39bd6f8f0fba36c772bb4f64f3d5a09566d8b8a5fe80b821c0cd4" exitCode=0 Oct 30 01:25:36.100790 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.100646 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" event={"ID":"2a68439c-dee6-4fb4-857d-bd09bbe8e214","Type":"ContainerDied","Data":"0cd3d38e51e39bd6f8f0fba36c772bb4f64f3d5a09566d8b8a5fe80b821c0cd4"} Oct 30 01:25:36.240880 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.240854 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:25:36.321725 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.321692 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d"] Oct 30 01:25:36.322532 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.322515 5300 cpu_manager.go:414] "containerMap: removing container" podUID="0957caa6-ff6b-4cf8-be88-717b6434a22f" containerName="main" Oct 30 01:25:36.322611 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.322535 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0957caa6-ff6b-4cf8-be88-717b6434a22f" containerName="main" Oct 30 01:25:36.322611 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.322559 5300 cpu_manager.go:414] "containerMap: removing container" podUID="bb602fa6-731e-4991-9504-7c743abb828b" containerName="main" Oct 30 01:25:36.322611 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.322564 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb602fa6-731e-4991-9504-7c743abb828b" containerName="main" Oct 30 01:25:36.322611 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.322580 5300 cpu_manager.go:414] "containerMap: removing container" podUID="2a68439c-dee6-4fb4-857d-bd09bbe8e214" containerName="istio-proxy" Oct 30 01:25:36.322611 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.322586 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a68439c-dee6-4fb4-857d-bd09bbe8e214" containerName="istio-proxy" Oct 30 01:25:36.322611 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.322592 5300 cpu_manager.go:414] "containerMap: removing container" podUID="ff735ea9-e315-46b8-b613-8164e6a2823a" containerName="main" Oct 30 01:25:36.322611 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.322597 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff735ea9-e315-46b8-b613-8164e6a2823a" containerName="main" Oct 30 01:25:36.322611 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.322612 5300 cpu_manager.go:414] "containerMap: removing container" podUID="3f4fe743-0164-4a05-a608-5d7078948f3a" containerName="main" Oct 30 01:25:36.322892 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.322618 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f4fe743-0164-4a05-a608-5d7078948f3a" containerName="main" Oct 30 01:25:36.322892 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.322805 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="0957caa6-ff6b-4cf8-be88-717b6434a22f" containerName="main" Oct 30 01:25:36.322892 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.322823 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb602fa6-731e-4991-9504-7c743abb828b" containerName="main" Oct 30 01:25:36.322892 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.322837 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a68439c-dee6-4fb4-857d-bd09bbe8e214" containerName="istio-proxy" Oct 30 01:25:36.322892 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.322845 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff735ea9-e315-46b8-b613-8164e6a2823a" containerName="main" Oct 30 01:25:36.322892 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.322858 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f4fe743-0164-4a05-a608-5d7078948f3a" containerName="main" Oct 30 01:25:36.335841 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.334590 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-data\") pod \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " Oct 30 01:25:36.335841 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.334659 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a68439c-dee6-4fb4-857d-bd09bbe8e214-kube-api-access-njzns\" (UniqueName: \"kubernetes.io/projected/2a68439c-dee6-4fb4-857d-bd09bbe8e214-kube-api-access-njzns\") pod \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " Oct 30 01:25:36.335841 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.334706 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-workload-socket\") pod \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " Oct 30 01:25:36.335841 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.334831 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istiod-ca-cert\") pod \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " Oct 30 01:25:36.335841 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.334906 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-credential-socket\") pod \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " Oct 30 01:25:36.335841 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.334936 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-token\" (UniqueName: \"kubernetes.io/projected/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-token\") pod \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " Oct 30 01:25:36.335841 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.335006 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-envoy\") pod \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " Oct 30 01:25:36.335841 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.335042 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-workload-certs\") pod \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " Oct 30 01:25:36.335841 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.335167 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-podinfo\") pod \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\" (UID: \"2a68439c-dee6-4fb4-857d-bd09bbe8e214\") " Oct 30 01:25:36.337097 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.336838 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-credential-socket" pod "2a68439c-dee6-4fb4-857d-bd09bbe8e214" (UID: "2a68439c-dee6-4fb4-857d-bd09bbe8e214"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:36.337097 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.336920 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istiod-ca-cert" pod "2a68439c-dee6-4fb4-857d-bd09bbe8e214" (UID: "2a68439c-dee6-4fb4-857d-bd09bbe8e214"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:25:36.337097 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.337033 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-data" pod "2a68439c-dee6-4fb4-857d-bd09bbe8e214" (UID: "2a68439c-dee6-4fb4-857d-bd09bbe8e214"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:36.343803 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.337625 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-workload-certs" pod "2a68439c-dee6-4fb4-857d-bd09bbe8e214" (UID: "2a68439c-dee6-4fb4-857d-bd09bbe8e214"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:36.343803 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.338075 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:25:36.343803 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.342232 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-workload-socket" pod "2a68439c-dee6-4fb4-857d-bd09bbe8e214" (UID: "2a68439c-dee6-4fb4-857d-bd09bbe8e214"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:36.344071 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.344042 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:25:36.344199 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.344078 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:25:36.344199 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.344093 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:25:36.349810 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.349790 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d"] Oct 30 01:25:36.350279 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.349936 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.355125 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.355103 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-envoy" pod "2a68439c-dee6-4fb4-857d-bd09bbe8e214" (UID: "2a68439c-dee6-4fb4-857d-bd09bbe8e214"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:25:36.355982 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.355960 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a68439c-dee6-4fb4-857d-bd09bbe8e214-kube-api-access-njzns" pod "2a68439c-dee6-4fb4-857d-bd09bbe8e214" (UID: "2a68439c-dee6-4fb4-857d-bd09bbe8e214"). InnerVolumeSpecName "kube-api-access-njzns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:36.357394 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.357366 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-podinfo" pod "2a68439c-dee6-4fb4-857d-bd09bbe8e214" (UID: "2a68439c-dee6-4fb4-857d-bd09bbe8e214"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:25:36.358975 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.358944 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-token" pod "2a68439c-dee6-4fb4-857d-bd09bbe8e214" (UID: "2a68439c-dee6-4fb4-857d-bd09bbe8e214"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:36.369109 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.369084 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-368bbb-openshift-default-dockercfg-6hk78\"" Oct 30 01:25:36.445884 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.445797 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-istio-data\") pod \"e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.446066 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.445948 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-workload-socket\") pod \"e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.446066 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.446012 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br656\" (UniqueName: \"kubernetes.io/projected/85d29953-787a-49a2-87d5-1514f1c425ee-kube-api-access-br656\") pod \"e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.446246 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.446224 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d29953-787a-49a2-87d5-1514f1c425ee-istio-token\") pod \"e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.446311 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.446282 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d29953-787a-49a2-87d5-1514f1c425ee-istio-podinfo\") pod \"e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.446355 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.446344 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-istio-envoy\") pod \"e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.446406 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.446395 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-workload-certs\") pod \"e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.446475 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.446460 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-credential-socket\") pod \"e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.446590 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.446577 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d29953-787a-49a2-87d5-1514f1c425ee-istiod-ca-cert\") pod \"e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.446700 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.446689 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:25:36.446836 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.446702 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:25:36.446836 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.446711 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a68439c-dee6-4fb4-857d-bd09bbe8e214-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:25:36.446836 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.446720 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njzns\" (UniqueName: \"kubernetes.io/projected/2a68439c-dee6-4fb4-857d-bd09bbe8e214-kube-api-access-njzns\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:25:36.446836 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.446729 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a68439c-dee6-4fb4-857d-bd09bbe8e214-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:25:36.548347 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.548310 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d29953-787a-49a2-87d5-1514f1c425ee-istiod-ca-cert\") pod \"e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.548520 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.548386 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-istio-data\") pod \"e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.548520 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.548471 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-workload-socket\") pod \"e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.548654 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.548535 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br656\" (UniqueName: \"kubernetes.io/projected/85d29953-787a-49a2-87d5-1514f1c425ee-kube-api-access-br656\") pod \"e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.548717 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.548683 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d29953-787a-49a2-87d5-1514f1c425ee-istio-token\") pod \"e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.548817 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.548727 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d29953-787a-49a2-87d5-1514f1c425ee-istio-podinfo\") pod \"e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.548817 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.548786 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-istio-envoy\") pod \"e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.548944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.548851 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-workload-certs\") pod \"e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.548944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.548890 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-credential-socket\") pod \"e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.548944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.548916 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-workload-socket\") pod \"e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.549269 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.549243 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d29953-787a-49a2-87d5-1514f1c425ee-istiod-ca-cert\") pod \"e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.549457 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.549280 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-credential-socket\") pod \"e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.549561 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.549325 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-workload-certs\") pod \"e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.549561 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.549400 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-istio-data\") pod \"e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.551720 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.551695 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d29953-787a-49a2-87d5-1514f1c425ee-istio-podinfo\") pod \"e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.552589 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.552565 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-istio-envoy\") pod \"e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.570595 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.570568 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d29953-787a-49a2-87d5-1514f1c425ee-istio-token\") pod \"e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.570750 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.570731 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br656\" (UniqueName: \"kubernetes.io/projected/85d29953-787a-49a2-87d5-1514f1c425ee-kube-api-access-br656\") pod \"e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:36.699385 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:36.699292 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:37.022987 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:25:37.022954 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d29953_787a_49a2_87d5_1514f1c425ee.slice/crio-0e828edd81cefabe499dcccd4382d5bad6c6b48f62e4fbefd5f1b4d7ce17ef79 WatchSource:0}: Error finding container 0e828edd81cefabe499dcccd4382d5bad6c6b48f62e4fbefd5f1b4d7ce17ef79: Status 404 returned error can't find the container with id 0e828edd81cefabe499dcccd4382d5bad6c6b48f62e4fbefd5f1b4d7ce17ef79 Oct 30 01:25:37.023959 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:37.023932 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d"] Oct 30 01:25:37.026674 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:37.026639 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:25:37.026777 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:37.026708 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:25:37.026777 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:37.026741 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:25:37.109750 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:37.109726 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" Oct 30 01:25:37.109914 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:37.109753 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq" event={"ID":"2a68439c-dee6-4fb4-857d-bd09bbe8e214","Type":"ContainerDied","Data":"2b259068ca24907ce5acebf5a2a744db6325d93520606300ca4e25fecb014309"} Oct 30 01:25:37.109914 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:37.109849 5300 scope.go:122] "RemoveContainer" containerID="0cd3d38e51e39bd6f8f0fba36c772bb4f64f3d5a09566d8b8a5fe80b821c0cd4" Oct 30 01:25:37.111974 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:37.111949 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" event={"ID":"85d29953-787a-49a2-87d5-1514f1c425ee","Type":"ContainerStarted","Data":"0e828edd81cefabe499dcccd4382d5bad6c6b48f62e4fbefd5f1b4d7ce17ef79"} Oct 30 01:25:37.169217 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:37.169184 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq"] Oct 30 01:25:37.174390 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:37.174365 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-eacea3-openshift-default-68bb98d95f-cfnpq"] Oct 30 01:25:37.469578 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:37.469494 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:37.825212 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:37.825182 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a68439c-dee6-4fb4-857d-bd09bbe8e214" path="/var/lib/kubelet/pods/2a68439c-dee6-4fb4-857d-bd09bbe8e214/volumes" Oct 30 01:25:38.122449 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:38.122348 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" event={"ID":"85d29953-787a-49a2-87d5-1514f1c425ee","Type":"ContainerStarted","Data":"941f4f0b7c5f816640828673b59e1bad24d087dfbfe2dc5a407cec115d803b8b"} Oct 30 01:25:38.184437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:38.184388 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" podStartSLOduration=68.184371925 podStartE2EDuration="1m8.184371925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:38.16746651 +0000 UTC m=+2538.483334836" watchObservedRunningTime="2026-10-30 01:25:38.184371925 +0000 UTC m=+2538.500240250" Oct 30 01:25:38.687845 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:38.687806 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:38.700065 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:38.699957 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:38.706593 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:38.706567 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:38.861242 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:38.861210 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:38.958458 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:38.958362 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:39.342984 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:39.338822 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:39.343528 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:39.343028 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:25:39.409301 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:39.409253 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:39.577030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:39.576992 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:39.660328 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:39.660241 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:40.065398 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:40.065367 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:40.928859 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:40.928819 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:41.038864 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:41.038832 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:41.417833 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:41.417806 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:42.658755 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:42.658720 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:42.760611 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:42.760578 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:43.080881 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:43.080849 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:44.929357 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:44.929326 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:45.835398 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:45.835367 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:45.942072 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:45.942037 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:46.172548 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:46.172462 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:46.812934 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:46.812882 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:48.616926 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:48.616867 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:50.164054 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:50.164021 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:51.595866 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:25:51.595824 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:25:51.719376 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:51.719345 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:53.260143 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:53.260115 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-nkqms_7817f490-6fc5-4d17-8acc-f83273bd9a5c/payload-processing/0.log" Oct 30 01:25:53.458888 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:53.458862 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-nkqms_7817f490-6fc5-4d17-8acc-f83273bd9a5c/payload-processing/0.log" Oct 30 01:25:53.784989 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:53.784960 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:55.990963 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:55.990934 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:57.482092 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:57.482060 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:25:58.994469 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:25:58.994438 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:00.693059 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:00.693024 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:02.596829 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:02.596789 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:04.108022 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:04.107992 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:05.639528 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:05.639498 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:07.107423 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:07.107389 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:08.554981 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:08.554948 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:10.238873 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:10.238845 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:11.532684 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:11.532644 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg"] Oct 30 01:26:11.554139 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:11.554094 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:11.713003 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:11.711401 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:13.306371 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:13.306331 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:14.739995 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:14.739959 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:16.182976 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:16.182942 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:17.702169 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:17.702104 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:19.337540 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:19.337505 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:20.791894 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:20.791862 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:21.594940 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:26:21.594898 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:26:22.196915 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:22.196878 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:23.181860 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:23.181820 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8"] Oct 30 01:26:23.214665 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:23.214626 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:23.841411 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:23.841375 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:23.927093 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:23.927060 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:24.392109 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:24.391960 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:24.681207 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:24.681101 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:25.638477 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:25.638436 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:25.735722 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:25.735694 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:25.981508 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:25.981460 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:26.558963 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:26.558923 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" podUID="d91d3868-2251-4508-b4d2-0efb0265293a" containerName="main" containerID="cri-o://b0766915757611ae2424740bc3ad7c96376e7e06c1d19efc6d884641f5a034f5" gracePeriod=45 Oct 30 01:26:26.804670 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:26.800028 5300 generic.go:356] "Generic (PLEG): container finished" podID="d91d3868-2251-4508-b4d2-0efb0265293a" containerID="b0766915757611ae2424740bc3ad7c96376e7e06c1d19efc6d884641f5a034f5" exitCode=0 Oct 30 01:26:26.804670 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:26.800302 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" event={"ID":"d91d3868-2251-4508-b4d2-0efb0265293a","Type":"ContainerDied","Data":"b0766915757611ae2424740bc3ad7c96376e7e06c1d19efc6d884641f5a034f5"} Oct 30 01:26:26.948193 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:26.948160 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" Oct 30 01:26:27.025467 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:27.025429 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d91d3868-2251-4508-b4d2-0efb0265293a-tls-certs\" (UniqueName: \"kubernetes.io/secret/d91d3868-2251-4508-b4d2-0efb0265293a-tls-certs\") pod \"d91d3868-2251-4508-b4d2-0efb0265293a\" (UID: \"d91d3868-2251-4508-b4d2-0efb0265293a\") " Oct 30 01:26:27.025911 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:27.025875 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-model-cache\") pod \"d91d3868-2251-4508-b4d2-0efb0265293a\" (UID: \"d91d3868-2251-4508-b4d2-0efb0265293a\") " Oct 30 01:26:27.026488 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:27.026468 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-tmp-dir\") pod \"d91d3868-2251-4508-b4d2-0efb0265293a\" (UID: \"d91d3868-2251-4508-b4d2-0efb0265293a\") " Oct 30 01:26:27.026955 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:27.026936 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-home\" (UniqueName: \"kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-home\") pod \"d91d3868-2251-4508-b4d2-0efb0265293a\" (UID: \"d91d3868-2251-4508-b4d2-0efb0265293a\") " Oct 30 01:26:27.027468 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:27.027448 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-dshm\") pod \"d91d3868-2251-4508-b4d2-0efb0265293a\" (UID: \"d91d3868-2251-4508-b4d2-0efb0265293a\") " Oct 30 01:26:27.028021 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:27.026311 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-model-cache" pod "d91d3868-2251-4508-b4d2-0efb0265293a" (UID: "d91d3868-2251-4508-b4d2-0efb0265293a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:27.028124 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:27.026825 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-tmp-dir" pod "d91d3868-2251-4508-b4d2-0efb0265293a" (UID: "d91d3868-2251-4508-b4d2-0efb0265293a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:27.028124 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:27.027273 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-home" pod "d91d3868-2251-4508-b4d2-0efb0265293a" (UID: "d91d3868-2251-4508-b4d2-0efb0265293a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:27.028422 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:27.028392 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d91d3868-2251-4508-b4d2-0efb0265293a-tls-certs" pod "d91d3868-2251-4508-b4d2-0efb0265293a" (UID: "d91d3868-2251-4508-b4d2-0efb0265293a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:26:27.028513 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:27.028407 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91d3868-2251-4508-b4d2-0efb0265293a-kube-api-access-rxz6g\" (UniqueName: \"kubernetes.io/projected/d91d3868-2251-4508-b4d2-0efb0265293a-kube-api-access-rxz6g\") pod \"d91d3868-2251-4508-b4d2-0efb0265293a\" (UID: \"d91d3868-2251-4508-b4d2-0efb0265293a\") " Oct 30 01:26:27.029382 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:27.029307 5300 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91d3868-2251-4508-b4d2-0efb0265293a-tls-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:26:27.029485 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:27.029388 5300 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-model-cache\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:26:27.029485 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:27.029428 5300 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-tmp-dir\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:26:27.029485 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:27.029440 5300 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-home\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:26:27.030338 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:27.030312 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-dshm" pod "d91d3868-2251-4508-b4d2-0efb0265293a" (UID: "d91d3868-2251-4508-b4d2-0efb0265293a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:27.030950 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:27.030925 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91d3868-2251-4508-b4d2-0efb0265293a-kube-api-access-rxz6g" pod "d91d3868-2251-4508-b4d2-0efb0265293a" (UID: "d91d3868-2251-4508-b4d2-0efb0265293a"). InnerVolumeSpecName "kube-api-access-rxz6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:27.131669 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:27.131633 5300 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91d3868-2251-4508-b4d2-0efb0265293a-dshm\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:26:27.131669 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:27.131668 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxz6g\" (UniqueName: \"kubernetes.io/projected/d91d3868-2251-4508-b4d2-0efb0265293a-kube-api-access-rxz6g\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:26:27.258014 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:27.257969 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:27.808512 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:27.808475 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" event={"ID":"d91d3868-2251-4508-b4d2-0efb0265293a","Type":"ContainerDied","Data":"ead750cffc8d4211b185831c3bb8e4163fcb341d88edecf3d87bc708dd2ad957"} Oct 30 01:26:27.808512 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:27.808504 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg" Oct 30 01:26:27.809071 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:27.808527 5300 scope.go:122] "RemoveContainer" containerID="b0766915757611ae2424740bc3ad7c96376e7e06c1d19efc6d884641f5a034f5" Oct 30 01:26:27.865965 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:27.865938 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg"] Oct 30 01:26:27.870511 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:27.870487 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-d81f0a/auth-test-model-d81f0a-kserve-7dd8cccb65-rqzlg"] Oct 30 01:26:28.845298 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:28.845260 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:29.829713 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:29.829678 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d91d3868-2251-4508-b4d2-0efb0265293a" path="/var/lib/kubelet/pods/d91d3868-2251-4508-b4d2-0efb0265293a/volumes" Oct 30 01:26:30.554736 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:30.554699 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:32.266424 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:32.266398 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:33.907131 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:33.907091 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:34.937564 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:34.937530 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:35.839231 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:35.839195 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:37.385059 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:37.385025 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:38.206067 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.206022 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" podUID="6e04fb5b-dba1-4e40-8a14-ff555398e2ee" containerName="main" containerID="cri-o://ec4c1dbecae5d8d2241248eb006dda85dbd880e34aee7c759155dc410d5aae96" gracePeriod=45 Oct 30 01:26:38.520309 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.520278 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" Oct 30 01:26:38.641225 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.641184 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-home\" (UniqueName: \"kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-home\") pod \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\" (UID: \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\") " Oct 30 01:26:38.641417 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.641256 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-model-cache\") pod \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\" (UID: \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\") " Oct 30 01:26:38.641417 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.641304 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-tmp-dir\") pod \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\" (UID: \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\") " Oct 30 01:26:38.641515 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.641448 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-tls-certs\" (UniqueName: \"kubernetes.io/secret/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-tls-certs\") pod \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\" (UID: \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\") " Oct 30 01:26:38.641678 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.641660 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-kube-api-access-jf9h9\" (UniqueName: \"kubernetes.io/projected/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-kube-api-access-jf9h9\") pod \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\" (UID: \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\") " Oct 30 01:26:38.641773 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.641723 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-dshm\") pod \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\" (UID: \"6e04fb5b-dba1-4e40-8a14-ff555398e2ee\") " Oct 30 01:26:38.643644 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.643613 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-model-cache" pod "6e04fb5b-dba1-4e40-8a14-ff555398e2ee" (UID: "6e04fb5b-dba1-4e40-8a14-ff555398e2ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:38.643884 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.643680 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-home" pod "6e04fb5b-dba1-4e40-8a14-ff555398e2ee" (UID: "6e04fb5b-dba1-4e40-8a14-ff555398e2ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:38.643884 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.643853 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-tmp-dir" pod "6e04fb5b-dba1-4e40-8a14-ff555398e2ee" (UID: "6e04fb5b-dba1-4e40-8a14-ff555398e2ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:38.648862 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.646165 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-dshm" pod "6e04fb5b-dba1-4e40-8a14-ff555398e2ee" (UID: "6e04fb5b-dba1-4e40-8a14-ff555398e2ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:38.648862 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.646644 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-kube-api-access-jf9h9" pod "6e04fb5b-dba1-4e40-8a14-ff555398e2ee" (UID: "6e04fb5b-dba1-4e40-8a14-ff555398e2ee"). InnerVolumeSpecName "kube-api-access-jf9h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:38.649676 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.649560 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-tls-certs" pod "6e04fb5b-dba1-4e40-8a14-ff555398e2ee" (UID: "6e04fb5b-dba1-4e40-8a14-ff555398e2ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:26:38.679894 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.679862 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:38.743826 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.743743 5300 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-tls-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:26:38.743826 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.743783 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf9h9\" (UniqueName: \"kubernetes.io/projected/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-kube-api-access-jf9h9\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:26:38.743826 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.743793 5300 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-dshm\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:26:38.743826 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.743802 5300 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-home\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:26:38.743826 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.743816 5300 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-model-cache\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:26:38.743826 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.743829 5300 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e04fb5b-dba1-4e40-8a14-ff555398e2ee-tmp-dir\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:26:38.837887 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.837851 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:38.909345 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.909290 5300 generic.go:356] "Generic (PLEG): container finished" podID="6e04fb5b-dba1-4e40-8a14-ff555398e2ee" containerID="ec4c1dbecae5d8d2241248eb006dda85dbd880e34aee7c759155dc410d5aae96" exitCode=0 Oct 30 01:26:38.909532 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.909354 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" Oct 30 01:26:38.909532 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.909518 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" event={"ID":"6e04fb5b-dba1-4e40-8a14-ff555398e2ee","Type":"ContainerDied","Data":"ec4c1dbecae5d8d2241248eb006dda85dbd880e34aee7c759155dc410d5aae96"} Oct 30 01:26:38.909669 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.909550 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8" event={"ID":"6e04fb5b-dba1-4e40-8a14-ff555398e2ee","Type":"ContainerDied","Data":"756db2bf48e1eae74e63d7d95074b36e8e1330c6bbf6ac58f0de8c56f5abb359"} Oct 30 01:26:38.909669 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.909566 5300 scope.go:122] "RemoveContainer" containerID="ec4c1dbecae5d8d2241248eb006dda85dbd880e34aee7c759155dc410d5aae96" Oct 30 01:26:38.918730 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.918704 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:38.922531 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.922510 5300 scope.go:122] "RemoveContainer" containerID="ec4c1dbecae5d8d2241248eb006dda85dbd880e34aee7c759155dc410d5aae96" Oct 30 01:26:38.922800 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:26:38.922751 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec4c1dbecae5d8d2241248eb006dda85dbd880e34aee7c759155dc410d5aae96\": container with ID starting with ec4c1dbecae5d8d2241248eb006dda85dbd880e34aee7c759155dc410d5aae96 not found: ID does not exist" containerID="ec4c1dbecae5d8d2241248eb006dda85dbd880e34aee7c759155dc410d5aae96" Oct 30 01:26:38.922855 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.922812 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec4c1dbecae5d8d2241248eb006dda85dbd880e34aee7c759155dc410d5aae96"} err="failed to get container status \"ec4c1dbecae5d8d2241248eb006dda85dbd880e34aee7c759155dc410d5aae96\": rpc error: code = NotFound desc = could not find container \"ec4c1dbecae5d8d2241248eb006dda85dbd880e34aee7c759155dc410d5aae96\": container with ID starting with ec4c1dbecae5d8d2241248eb006dda85dbd880e34aee7c759155dc410d5aae96 not found: ID does not exist" Oct 30 01:26:38.967687 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.967656 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8"] Oct 30 01:26:38.972593 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:38.972566 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-e92648/auth-test-model-e92648-kserve-549f496b8f-hkmw8"] Oct 30 01:26:39.303383 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:39.303351 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:39.466390 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:39.466351 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:39.572167 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:39.572070 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:39.831196 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:39.826459 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e04fb5b-dba1-4e40-8a14-ff555398e2ee" path="/var/lib/kubelet/pods/6e04fb5b-dba1-4e40-8a14-ff555398e2ee/volumes" Oct 30 01:26:40.014697 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:40.014664 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:41.356419 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:41.356375 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:43.033615 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:43.033533 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:43.142567 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:43.142529 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:44.791379 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:44.791336 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:44.891602 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:44.891562 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:44.991676 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:44.991650 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:46.064351 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:46.064316 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:46.154606 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:46.154574 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:46.435361 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:46.435264 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666"] Oct 30 01:26:46.436381 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:46.436348 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" podUID="94d4bbf4-260b-4a7e-bf75-a16c37b7e526" containerName="istio-proxy" containerID="cri-o://b2420f140b7665153a28c200bb2423adad3c92534cae5abaea824eddcb4d16b6" gracePeriod=30 Oct 30 01:26:46.473667 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:46.473640 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:47.297076 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:47.297045 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:48.024924 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:48.024875 5300 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-206.ec2.internal" podUID="3b175f0644888f98ec9cdc35355eb1b8" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:26:49.022537 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:49.022500 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:50.777652 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:50.777551 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:51.595347 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:26:51.595310 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:26:51.758558 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.758490 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:26:51.771619 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.771598 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-kube-api-access-967m4\" (UniqueName: \"kubernetes.io/projected/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-kube-api-access-967m4\") pod \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " Oct 30 01:26:51.771749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.771658 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-envoy\") pod \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " Oct 30 01:26:51.771749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.771679 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-data\") pod \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " Oct 30 01:26:51.771854 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.771804 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-podinfo\") pod \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " Oct 30 01:26:51.771895 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.771869 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istiod-ca-cert\") pod \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " Oct 30 01:26:51.771952 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.771895 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-workload-socket\") pod \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " Oct 30 01:26:51.772011 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.771988 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-token\" (UniqueName: \"kubernetes.io/projected/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-token\") pod \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " Oct 30 01:26:51.772068 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.772030 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-workload-certs\") pod \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " Oct 30 01:26:51.772128 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.772063 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-data" pod "94d4bbf4-260b-4a7e-bf75-a16c37b7e526" (UID: "94d4bbf4-260b-4a7e-bf75-a16c37b7e526"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:51.772192 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.772143 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-credential-socket\") pod \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\" (UID: \"94d4bbf4-260b-4a7e-bf75-a16c37b7e526\") " Oct 30 01:26:51.772251 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.772208 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-workload-socket" pod "94d4bbf4-260b-4a7e-bf75-a16c37b7e526" (UID: "94d4bbf4-260b-4a7e-bf75-a16c37b7e526"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:51.772360 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.772333 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-workload-certs" pod "94d4bbf4-260b-4a7e-bf75-a16c37b7e526" (UID: "94d4bbf4-260b-4a7e-bf75-a16c37b7e526"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:51.772536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.772511 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-credential-socket" pod "94d4bbf4-260b-4a7e-bf75-a16c37b7e526" (UID: "94d4bbf4-260b-4a7e-bf75-a16c37b7e526"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:51.772627 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.772563 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istiod-ca-cert" pod "94d4bbf4-260b-4a7e-bf75-a16c37b7e526" (UID: "94d4bbf4-260b-4a7e-bf75-a16c37b7e526"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:26:51.772960 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.772941 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:26:51.772960 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.772959 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:26:51.773131 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.772968 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:26:51.773131 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.772977 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:26:51.773131 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.772985 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:26:51.774074 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.774053 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-kube-api-access-967m4" pod "94d4bbf4-260b-4a7e-bf75-a16c37b7e526" (UID: "94d4bbf4-260b-4a7e-bf75-a16c37b7e526"). InnerVolumeSpecName "kube-api-access-967m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:51.774237 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.774221 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-podinfo" pod "94d4bbf4-260b-4a7e-bf75-a16c37b7e526" (UID: "94d4bbf4-260b-4a7e-bf75-a16c37b7e526"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:26:51.774445 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.774423 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-envoy" pod "94d4bbf4-260b-4a7e-bf75-a16c37b7e526" (UID: "94d4bbf4-260b-4a7e-bf75-a16c37b7e526"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:26:51.774485 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.774442 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-token" pod "94d4bbf4-260b-4a7e-bf75-a16c37b7e526" (UID: "94d4bbf4-260b-4a7e-bf75-a16c37b7e526"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:26:51.862950 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.862915 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq"] Oct 30 01:26:51.864696 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.864669 5300 cpu_manager.go:414] "containerMap: removing container" podUID="6e04fb5b-dba1-4e40-8a14-ff555398e2ee" containerName="main" Oct 30 01:26:51.864696 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.864696 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e04fb5b-dba1-4e40-8a14-ff555398e2ee" containerName="main" Oct 30 01:26:51.864902 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.864715 5300 cpu_manager.go:414] "containerMap: removing container" podUID="94d4bbf4-260b-4a7e-bf75-a16c37b7e526" containerName="istio-proxy" Oct 30 01:26:51.864902 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.864724 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d4bbf4-260b-4a7e-bf75-a16c37b7e526" containerName="istio-proxy" Oct 30 01:26:51.864902 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.864793 5300 cpu_manager.go:414] "containerMap: removing container" podUID="d91d3868-2251-4508-b4d2-0efb0265293a" containerName="main" Oct 30 01:26:51.864902 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.864802 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d91d3868-2251-4508-b4d2-0efb0265293a" containerName="main" Oct 30 01:26:51.865168 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.865066 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="d91d3868-2251-4508-b4d2-0efb0265293a" containerName="main" Oct 30 01:26:51.865168 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.865098 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e04fb5b-dba1-4e40-8a14-ff555398e2ee" containerName="main" Oct 30 01:26:51.865168 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.865108 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d4bbf4-260b-4a7e-bf75-a16c37b7e526" containerName="istio-proxy" Oct 30 01:26:51.870175 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.870153 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq" Oct 30 01:26:51.875474 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.875332 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-967m4\" (UniqueName: \"kubernetes.io/projected/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-kube-api-access-967m4\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:26:51.875474 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.875361 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:26:51.875474 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.875377 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:26:51.875474 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.875389 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d4bbf4-260b-4a7e-bf75-a16c37b7e526-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:26:51.886250 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.886198 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq"] Oct 30 01:26:51.889590 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.889567 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-29bb56-dockercfg-xvvrz\"" Oct 30 01:26:51.889699 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.889613 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-29bb56\"" Oct 30 01:26:51.977034 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.976993 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w4nj\" (UniqueName: \"kubernetes.io/projected/3f305e4e-92ce-4b37-adb8-3a8106e3854e-kube-api-access-6w4nj\") pod \"payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq\" (UID: \"3f305e4e-92ce-4b37-adb8-3a8106e3854e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq" Oct 30 01:26:51.977219 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:51.977196 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f305e4e-92ce-4b37-adb8-3a8106e3854e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq\" (UID: \"3f305e4e-92ce-4b37-adb8-3a8106e3854e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq" Oct 30 01:26:52.020356 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:52.020281 5300 generic.go:356] "Generic (PLEG): container finished" podID="94d4bbf4-260b-4a7e-bf75-a16c37b7e526" containerID="b2420f140b7665153a28c200bb2423adad3c92534cae5abaea824eddcb4d16b6" exitCode=0 Oct 30 01:26:52.020356 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:52.020347 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" Oct 30 01:26:52.020536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:52.020365 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" event={"ID":"94d4bbf4-260b-4a7e-bf75-a16c37b7e526","Type":"ContainerDied","Data":"b2420f140b7665153a28c200bb2423adad3c92534cae5abaea824eddcb4d16b6"} Oct 30 01:26:52.020536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:52.020441 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666" event={"ID":"94d4bbf4-260b-4a7e-bf75-a16c37b7e526","Type":"ContainerDied","Data":"226e5d590ac90b3d8e3ad53c4f6c5520d401e336ce95a450439e2224796aa5cf"} Oct 30 01:26:52.020536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:52.020465 5300 scope.go:122] "RemoveContainer" containerID="b2420f140b7665153a28c200bb2423adad3c92534cae5abaea824eddcb4d16b6" Oct 30 01:26:52.031480 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:52.031461 5300 scope.go:122] "RemoveContainer" containerID="b2420f140b7665153a28c200bb2423adad3c92534cae5abaea824eddcb4d16b6" Oct 30 01:26:52.031794 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:26:52.031750 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2420f140b7665153a28c200bb2423adad3c92534cae5abaea824eddcb4d16b6\": container with ID starting with b2420f140b7665153a28c200bb2423adad3c92534cae5abaea824eddcb4d16b6 not found: ID does not exist" containerID="b2420f140b7665153a28c200bb2423adad3c92534cae5abaea824eddcb4d16b6" Oct 30 01:26:52.031871 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:52.031792 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2420f140b7665153a28c200bb2423adad3c92534cae5abaea824eddcb4d16b6"} err="failed to get container status \"b2420f140b7665153a28c200bb2423adad3c92534cae5abaea824eddcb4d16b6\": rpc error: code = NotFound desc = could not find container \"b2420f140b7665153a28c200bb2423adad3c92534cae5abaea824eddcb4d16b6\": container with ID starting with b2420f140b7665153a28c200bb2423adad3c92534cae5abaea824eddcb4d16b6 not found: ID does not exist" Oct 30 01:26:52.078916 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:52.078881 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w4nj\" (UniqueName: \"kubernetes.io/projected/3f305e4e-92ce-4b37-adb8-3a8106e3854e-kube-api-access-6w4nj\") pod \"payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq\" (UID: \"3f305e4e-92ce-4b37-adb8-3a8106e3854e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq" Oct 30 01:26:52.079079 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:52.079065 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666"] Oct 30 01:26:52.079156 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:52.079132 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f305e4e-92ce-4b37-adb8-3a8106e3854e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq\" (UID: \"3f305e4e-92ce-4b37-adb8-3a8106e3854e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq" Oct 30 01:26:52.079885 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:52.079863 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f305e4e-92ce-4b37-adb8-3a8106e3854e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq\" (UID: \"3f305e4e-92ce-4b37-adb8-3a8106e3854e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq" Oct 30 01:26:52.105102 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:52.105070 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2a05c-openshift-default-dcbdf5575-kf666"] Oct 30 01:26:52.105562 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:52.105540 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w4nj\" (UniqueName: \"kubernetes.io/projected/3f305e4e-92ce-4b37-adb8-3a8106e3854e-kube-api-access-6w4nj\") pod \"payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq\" (UID: \"3f305e4e-92ce-4b37-adb8-3a8106e3854e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq" Oct 30 01:26:52.194816 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:52.194756 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq" Oct 30 01:26:52.504305 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:26:52.504269 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f305e4e_92ce_4b37_adb8_3a8106e3854e.slice/crio-5b560ed9cba13351b5fe2ba765eee1f3494f8e08b896dcc1917cf7e5bb9e09ea WatchSource:0}: Error finding container 5b560ed9cba13351b5fe2ba765eee1f3494f8e08b896dcc1917cf7e5bb9e09ea: Status 404 returned error can't find the container with id 5b560ed9cba13351b5fe2ba765eee1f3494f8e08b896dcc1917cf7e5bb9e09ea Oct 30 01:26:52.504745 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:52.504682 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq"] Oct 30 01:26:52.631830 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:52.631792 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:53.033165 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:53.033125 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq" event={"ID":"3f305e4e-92ce-4b37-adb8-3a8106e3854e","Type":"ContainerStarted","Data":"7820ee8e3de2da43875efe602c7eeec4007d333a319ef415658b27eb149e4393"} Oct 30 01:26:53.033651 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:53.033183 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq" event={"ID":"3f305e4e-92ce-4b37-adb8-3a8106e3854e","Type":"ContainerStarted","Data":"5b560ed9cba13351b5fe2ba765eee1f3494f8e08b896dcc1917cf7e5bb9e09ea"} Oct 30 01:26:53.033651 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:53.033220 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq" Oct 30 01:26:53.077036 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:53.076986 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq" podStartSLOduration=8.076972268 podStartE2EDuration="8.076972268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:53.073597845 +0000 UTC m=+2613.389466171" watchObservedRunningTime="2026-10-30 01:26:53.076972268 +0000 UTC m=+2613.392840903" Oct 30 01:26:53.825445 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:53.825410 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94d4bbf4-260b-4a7e-bf75-a16c37b7e526" path="/var/lib/kubelet/pods/94d4bbf4-260b-4a7e-bf75-a16c37b7e526/volumes" Oct 30 01:26:54.169592 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:54.169503 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:55.730062 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:55.730026 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:57.149357 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:57.149323 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:57.591678 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:57.591632 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq"] Oct 30 01:26:57.635893 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:57.635863 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l"] Oct 30 01:26:57.732039 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:57.732009 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:57.832744 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:57.832714 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:26:58.838348 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:26:58.838314 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:00.421239 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:00.421204 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:00.829846 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:00.829754 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:00.926193 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:00.926159 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:01.825775 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:01.825738 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:02.592900 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:02.592860 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq" podUID="43d688cb-3c2f-4d2d-84d6-1dbadcdb2985" containerName="payload-processing" containerID="cri-o://024b06f5187ad8ee39f85879f1cb251b25cebd771f9cb7648dcaa366283cd0e1" gracePeriod=25 Oct 30 01:27:02.636962 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:02.636925 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l" podUID="f667d6d6-df98-49e9-9d07-c9ae6e692863" containerName="payload-pre-processing" containerID="cri-o://14e04700662f3a2cf006ee4e72e48182b4f1e30f2397ea3944707611de7ba3bd" gracePeriod=25 Oct 30 01:27:03.492561 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:03.492530 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:04.044434 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:04.044406 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq" Oct 30 01:27:05.027531 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:05.027500 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:06.261256 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:06.261226 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:06.883742 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:06.883712 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:06.986574 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:06.986545 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:07.103015 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:07.102982 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:07.523580 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:07.523540 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:08.035045 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:08.035006 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l" podUID="f667d6d6-df98-49e9-9d07-c9ae6e692863" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:27:08.035234 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:08.035059 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq" podUID="43d688cb-3c2f-4d2d-84d6-1dbadcdb2985" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:27:08.668423 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:08.668393 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:10.447140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:10.447096 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:11.098788 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:11.098730 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd"] Oct 30 01:27:11.189959 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:11.189917 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:11.286106 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:11.286061 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:12.090147 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:12.090109 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:13.143869 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:13.142721 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:13.749104 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:13.749060 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:13.845186 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:13.845153 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:14.112962 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:14.112876 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:14.211324 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:14.211296 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:14.735376 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:14.735341 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:15.407986 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:15.407957 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:15.645673 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:15.645638 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:15.752946 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:15.752905 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:15.870102 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:15.870073 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:16.099639 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:16.099557 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd" podUID="5ba6e8c3-2254-495b-9f97-55fba6c120c8" containerName="payload-pre-processing" containerID="cri-o://e77f3db2491f6508515e835df4899884cc275247b7f0c99f50d1571682e7a2aa" gracePeriod=25 Oct 30 01:27:16.607711 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:16.607538 5300 generic.go:356] "Generic (PLEG): container finished" podID="5ba6e8c3-2254-495b-9f97-55fba6c120c8" containerID="e77f3db2491f6508515e835df4899884cc275247b7f0c99f50d1571682e7a2aa" exitCode=0 Oct 30 01:27:16.607711 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:16.607646 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd" event={"ID":"5ba6e8c3-2254-495b-9f97-55fba6c120c8","Type":"ContainerDied","Data":"e77f3db2491f6508515e835df4899884cc275247b7f0c99f50d1571682e7a2aa"} Oct 30 01:27:16.699482 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:16.699458 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd" Oct 30 01:27:16.767747 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:16.767710 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba6e8c3-2254-495b-9f97-55fba6c120c8-kube-api-access-hsbdb\" (UniqueName: \"kubernetes.io/projected/5ba6e8c3-2254-495b-9f97-55fba6c120c8-kube-api-access-hsbdb\") pod \"5ba6e8c3-2254-495b-9f97-55fba6c120c8\" (UID: \"5ba6e8c3-2254-495b-9f97-55fba6c120c8\") " Oct 30 01:27:16.767930 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:16.767907 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ba6e8c3-2254-495b-9f97-55fba6c120c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ba6e8c3-2254-495b-9f97-55fba6c120c8-plugins-config-volume\") pod \"5ba6e8c3-2254-495b-9f97-55fba6c120c8\" (UID: \"5ba6e8c3-2254-495b-9f97-55fba6c120c8\") " Oct 30 01:27:16.768261 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:16.768236 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba6e8c3-2254-495b-9f97-55fba6c120c8-plugins-config-volume" pod "5ba6e8c3-2254-495b-9f97-55fba6c120c8" (UID: "5ba6e8c3-2254-495b-9f97-55fba6c120c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:16.768976 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:16.768951 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ba6e8c3-2254-495b-9f97-55fba6c120c8-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:27:16.769894 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:16.769872 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba6e8c3-2254-495b-9f97-55fba6c120c8-kube-api-access-hsbdb" pod "5ba6e8c3-2254-495b-9f97-55fba6c120c8" (UID: "5ba6e8c3-2254-495b-9f97-55fba6c120c8"). InnerVolumeSpecName "kube-api-access-hsbdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:16.870530 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:16.870498 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsbdb\" (UniqueName: \"kubernetes.io/projected/5ba6e8c3-2254-495b-9f97-55fba6c120c8-kube-api-access-hsbdb\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:27:17.464926 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:17.464881 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:17.621693 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:17.621658 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd" event={"ID":"5ba6e8c3-2254-495b-9f97-55fba6c120c8","Type":"ContainerDied","Data":"e5d8722b509b75c647b1d7cb31dd4f513f1b895174870b97d1ade7642b15ab66"} Oct 30 01:27:17.622756 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:17.622734 5300 scope.go:122] "RemoveContainer" containerID="e77f3db2491f6508515e835df4899884cc275247b7f0c99f50d1571682e7a2aa" Oct 30 01:27:17.623069 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:17.622625 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd" Oct 30 01:27:17.698769 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:17.698738 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd"] Oct 30 01:27:17.712594 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:17.712562 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a6b650-7f4b88d675-r87zd"] Oct 30 01:27:17.848580 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:17.848547 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ba6e8c3-2254-495b-9f97-55fba6c120c8" path="/var/lib/kubelet/pods/5ba6e8c3-2254-495b-9f97-55fba6c120c8/volumes" Oct 30 01:27:18.043980 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:18.043935 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l" podUID="f667d6d6-df98-49e9-9d07-c9ae6e692863" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:27:18.045028 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:18.044986 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq" podUID="43d688cb-3c2f-4d2d-84d6-1dbadcdb2985" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:27:19.128846 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:19.128807 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:20.723254 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:20.723221 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:21.602334 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:27:21.602291 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:27:22.597480 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:22.597445 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.004025 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.008073 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.018093 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f667d6d6-df98-49e9-9d07-c9ae6e692863-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f667d6d6-df98-49e9-9d07-c9ae6e692863-plugins-config-volume\") pod \"f667d6d6-df98-49e9-9d07-c9ae6e692863\" (UID: \"f667d6d6-df98-49e9-9d07-c9ae6e692863\") " Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.018215 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f667d6d6-df98-49e9-9d07-c9ae6e692863-kube-api-access-ftgjs\" (UniqueName: \"kubernetes.io/projected/f667d6d6-df98-49e9-9d07-c9ae6e692863-kube-api-access-ftgjs\") pod \"f667d6d6-df98-49e9-9d07-c9ae6e692863\" (UID: \"f667d6d6-df98-49e9-9d07-c9ae6e692863\") " Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.018363 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43d688cb-3c2f-4d2d-84d6-1dbadcdb2985-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d688cb-3c2f-4d2d-84d6-1dbadcdb2985-plugins-config-volume\") pod \"43d688cb-3c2f-4d2d-84d6-1dbadcdb2985\" (UID: \"43d688cb-3c2f-4d2d-84d6-1dbadcdb2985\") " Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.018386 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43d688cb-3c2f-4d2d-84d6-1dbadcdb2985-kube-api-access-7fj95\" (UniqueName: \"kubernetes.io/projected/43d688cb-3c2f-4d2d-84d6-1dbadcdb2985-kube-api-access-7fj95\") pod \"43d688cb-3c2f-4d2d-84d6-1dbadcdb2985\" (UID: \"43d688cb-3c2f-4d2d-84d6-1dbadcdb2985\") " Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.018429 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f667d6d6-df98-49e9-9d07-c9ae6e692863-plugins-config-volume" pod "f667d6d6-df98-49e9-9d07-c9ae6e692863" (UID: "f667d6d6-df98-49e9-9d07-c9ae6e692863"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.018696 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d688cb-3c2f-4d2d-84d6-1dbadcdb2985-plugins-config-volume" pod "43d688cb-3c2f-4d2d-84d6-1dbadcdb2985" (UID: "43d688cb-3c2f-4d2d-84d6-1dbadcdb2985"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.018970 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d688cb-3c2f-4d2d-84d6-1dbadcdb2985-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.018984 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f667d6d6-df98-49e9-9d07-c9ae6e692863-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.020329 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f667d6d6-df98-49e9-9d07-c9ae6e692863-kube-api-access-ftgjs" pod "f667d6d6-df98-49e9-9d07-c9ae6e692863" (UID: "f667d6d6-df98-49e9-9d07-c9ae6e692863"). InnerVolumeSpecName "kube-api-access-ftgjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.020610 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d688cb-3c2f-4d2d-84d6-1dbadcdb2985-kube-api-access-7fj95" pod "43d688cb-3c2f-4d2d-84d6-1dbadcdb2985" (UID: "43d688cb-3c2f-4d2d-84d6-1dbadcdb2985"). InnerVolumeSpecName "kube-api-access-7fj95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.072723 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6"] Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.074000 5300 cpu_manager.go:414] "containerMap: removing container" podUID="5ba6e8c3-2254-495b-9f97-55fba6c120c8" containerName="payload-pre-processing" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.074022 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba6e8c3-2254-495b-9f97-55fba6c120c8" containerName="payload-pre-processing" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.074044 5300 cpu_manager.go:414] "containerMap: removing container" podUID="43d688cb-3c2f-4d2d-84d6-1dbadcdb2985" containerName="payload-processing" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.074052 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43d688cb-3c2f-4d2d-84d6-1dbadcdb2985" containerName="payload-processing" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.074069 5300 cpu_manager.go:414] "containerMap: removing container" podUID="f667d6d6-df98-49e9-9d07-c9ae6e692863" containerName="payload-pre-processing" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.074077 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f667d6d6-df98-49e9-9d07-c9ae6e692863" containerName="payload-pre-processing" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.074316 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba6e8c3-2254-495b-9f97-55fba6c120c8" containerName="payload-pre-processing" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.074348 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="43d688cb-3c2f-4d2d-84d6-1dbadcdb2985" containerName="payload-processing" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.074360 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="f667d6d6-df98-49e9-9d07-c9ae6e692863" containerName="payload-pre-processing" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.079015 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.097561 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-d81f0a-openshift-default-dockercfg-447gk\"" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.112968 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6"] Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.120734 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-data\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.120804 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4nfg\" (UniqueName: \"kubernetes.io/projected/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-kube-api-access-s4nfg\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.120910 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-credential-socket\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.121078 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istiod-ca-cert\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.121124 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-token\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.121195 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-workload-certs\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.121245 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-podinfo\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.121441 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-envoy\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.121472 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-workload-socket\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.121630 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftgjs\" (UniqueName: \"kubernetes.io/projected/f667d6d6-df98-49e9-9d07-c9ae6e692863-kube-api-access-ftgjs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.121649 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fj95\" (UniqueName: \"kubernetes.io/projected/43d688cb-3c2f-4d2d-84d6-1dbadcdb2985-kube-api-access-7fj95\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.222579 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istiod-ca-cert\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.222624 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-token\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.222644 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-workload-certs\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.222676 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-podinfo\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.222821 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-envoy\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.222851 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-workload-socket\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.222976 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-data\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.223013 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4nfg\" (UniqueName: \"kubernetes.io/projected/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-kube-api-access-s4nfg\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.223072 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-credential-socket\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.223113 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-workload-certs\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.223442 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istiod-ca-cert\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.224009 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-data\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.223986 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-workload-socket\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.224161 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-credential-socket\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.225060 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-envoy\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.226805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.225388 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-podinfo\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.246237 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.246168 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-token\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.247282 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.247262 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4nfg\" (UniqueName: \"kubernetes.io/projected/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-kube-api-access-s4nfg\") pod \"e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.400899 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.400856 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:23.538094 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.538009 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:23.643693 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.643656 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:23.678570 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.678535 5300 generic.go:356] "Generic (PLEG): container finished" podID="f667d6d6-df98-49e9-9d07-c9ae6e692863" containerID="14e04700662f3a2cf006ee4e72e48182b4f1e30f2397ea3944707611de7ba3bd" exitCode=0 Oct 30 01:27:23.678730 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.678576 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l" Oct 30 01:27:23.678730 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.678621 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l" event={"ID":"f667d6d6-df98-49e9-9d07-c9ae6e692863","Type":"ContainerDied","Data":"14e04700662f3a2cf006ee4e72e48182b4f1e30f2397ea3944707611de7ba3bd"} Oct 30 01:27:23.678730 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.678671 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l" event={"ID":"f667d6d6-df98-49e9-9d07-c9ae6e692863","Type":"ContainerDied","Data":"25979929eba14421c6d76f822a5f5460a0d409606307cfcc9a03c5473307bc1d"} Oct 30 01:27:23.678730 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.678696 5300 scope.go:122] "RemoveContainer" containerID="14e04700662f3a2cf006ee4e72e48182b4f1e30f2397ea3944707611de7ba3bd" Oct 30 01:27:23.680334 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.680309 5300 generic.go:356] "Generic (PLEG): container finished" podID="43d688cb-3c2f-4d2d-84d6-1dbadcdb2985" containerID="024b06f5187ad8ee39f85879f1cb251b25cebd771f9cb7648dcaa366283cd0e1" exitCode=0 Oct 30 01:27:23.680465 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.680340 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq" event={"ID":"43d688cb-3c2f-4d2d-84d6-1dbadcdb2985","Type":"ContainerDied","Data":"024b06f5187ad8ee39f85879f1cb251b25cebd771f9cb7648dcaa366283cd0e1"} Oct 30 01:27:23.680465 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.680361 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq" Oct 30 01:27:23.680465 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.680382 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq" event={"ID":"43d688cb-3c2f-4d2d-84d6-1dbadcdb2985","Type":"ContainerDied","Data":"824a7a2049a63de634ca537984284b8dac62f689c63aee462e38f2f9772cabf2"} Oct 30 01:27:23.682219 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.682197 5300 generic.go:356] "Generic (PLEG): container finished" podID="85d29953-787a-49a2-87d5-1514f1c425ee" containerID="941f4f0b7c5f816640828673b59e1bad24d087dfbfe2dc5a407cec115d803b8b" exitCode=0 Oct 30 01:27:23.682339 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.682277 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" event={"ID":"85d29953-787a-49a2-87d5-1514f1c425ee","Type":"ContainerDied","Data":"941f4f0b7c5f816640828673b59e1bad24d087dfbfe2dc5a407cec115d803b8b"} Oct 30 01:27:23.682733 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.682718 5300 scope.go:122] "RemoveContainer" containerID="941f4f0b7c5f816640828673b59e1bad24d087dfbfe2dc5a407cec115d803b8b" Oct 30 01:27:23.684983 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.684952 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:27:23.685084 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.685042 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:27:23.685176 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.685096 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:27:23.691380 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.691363 5300 scope.go:122] "RemoveContainer" containerID="14e04700662f3a2cf006ee4e72e48182b4f1e30f2397ea3944707611de7ba3bd" Oct 30 01:27:23.693409 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:27:23.693389 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14e04700662f3a2cf006ee4e72e48182b4f1e30f2397ea3944707611de7ba3bd\": container with ID starting with 14e04700662f3a2cf006ee4e72e48182b4f1e30f2397ea3944707611de7ba3bd not found: ID does not exist" containerID="14e04700662f3a2cf006ee4e72e48182b4f1e30f2397ea3944707611de7ba3bd" Oct 30 01:27:23.693512 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.693419 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14e04700662f3a2cf006ee4e72e48182b4f1e30f2397ea3944707611de7ba3bd"} err="failed to get container status \"14e04700662f3a2cf006ee4e72e48182b4f1e30f2397ea3944707611de7ba3bd\": rpc error: code = NotFound desc = could not find container \"14e04700662f3a2cf006ee4e72e48182b4f1e30f2397ea3944707611de7ba3bd\": container with ID starting with 14e04700662f3a2cf006ee4e72e48182b4f1e30f2397ea3944707611de7ba3bd not found: ID does not exist" Oct 30 01:27:23.693512 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.693442 5300 scope.go:122] "RemoveContainer" containerID="024b06f5187ad8ee39f85879f1cb251b25cebd771f9cb7648dcaa366283cd0e1" Oct 30 01:27:23.704325 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.704302 5300 scope.go:122] "RemoveContainer" containerID="024b06f5187ad8ee39f85879f1cb251b25cebd771f9cb7648dcaa366283cd0e1" Oct 30 01:27:23.704613 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:27:23.704587 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"024b06f5187ad8ee39f85879f1cb251b25cebd771f9cb7648dcaa366283cd0e1\": container with ID starting with 024b06f5187ad8ee39f85879f1cb251b25cebd771f9cb7648dcaa366283cd0e1 not found: ID does not exist" containerID="024b06f5187ad8ee39f85879f1cb251b25cebd771f9cb7648dcaa366283cd0e1" Oct 30 01:27:23.704718 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.704622 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"024b06f5187ad8ee39f85879f1cb251b25cebd771f9cb7648dcaa366283cd0e1"} err="failed to get container status \"024b06f5187ad8ee39f85879f1cb251b25cebd771f9cb7648dcaa366283cd0e1\": rpc error: code = NotFound desc = could not find container \"024b06f5187ad8ee39f85879f1cb251b25cebd771f9cb7648dcaa366283cd0e1\": container with ID starting with 024b06f5187ad8ee39f85879f1cb251b25cebd771f9cb7648dcaa366283cd0e1 not found: ID does not exist" Oct 30 01:27:23.730320 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:27:23.730294 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79971c3_5d7b_4fd1_a84f_46b2c0190af6.slice/crio-9717da3ccbd6e555b97ba0c27e0f476a922dba63a7d82a41e285a042d0fbae4d WatchSource:0}: Error finding container 9717da3ccbd6e555b97ba0c27e0f476a922dba63a7d82a41e285a042d0fbae4d: Status 404 returned error can't find the container with id 9717da3ccbd6e555b97ba0c27e0f476a922dba63a7d82a41e285a042d0fbae4d Oct 30 01:27:23.733184 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.733161 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6"] Oct 30 01:27:23.734253 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.734220 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:27:23.734359 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.734288 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:27:23.734359 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.734322 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:27:23.756171 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.756147 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l"] Oct 30 01:27:23.762060 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.761276 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-eacea3-6579cb8b7-cpm9l"] Oct 30 01:27:23.847983 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.845834 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f667d6d6-df98-49e9-9d07-c9ae6e692863" path="/var/lib/kubelet/pods/f667d6d6-df98-49e9-9d07-c9ae6e692863/volumes" Oct 30 01:27:23.859396 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.859366 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq"] Oct 30 01:27:23.863187 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:23.863162 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-eacea3-5dddf77dd-2lddq"] Oct 30 01:27:24.312127 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:24.312091 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:24.339141 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:24.339101 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:27:24.527231 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:24.527199 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:24.666553 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:24.666458 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:24.704363 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:24.704323 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" event={"ID":"f79971c3-5d7b-4fd1-a84f-46b2c0190af6","Type":"ContainerStarted","Data":"1a5d65f474e166141d53f1499262266ca0ca1fddece615a60b1b2b9a8a1c8945"} Oct 30 01:27:24.706578 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:24.706541 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" event={"ID":"f79971c3-5d7b-4fd1-a84f-46b2c0190af6","Type":"ContainerStarted","Data":"9717da3ccbd6e555b97ba0c27e0f476a922dba63a7d82a41e285a042d0fbae4d"} Oct 30 01:27:24.710701 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:24.710669 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" event={"ID":"85d29953-787a-49a2-87d5-1514f1c425ee","Type":"ContainerStarted","Data":"a12c568c8faf5863370778a422fd81d0ad39bbb2cad6c9c404a8b0fdf8a94bcc"} Oct 30 01:27:24.784506 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:24.784229 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" podStartSLOduration=210.784210665 podStartE2EDuration="3m30.784210665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:24.780058394 +0000 UTC m=+2645.095926748" watchObservedRunningTime="2026-10-30 01:27:24.784210665 +0000 UTC m=+2645.100079003" Oct 30 01:27:25.182898 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:25.177755 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb"] Oct 30 01:27:25.197192 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:25.197157 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:25.409203 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:25.409173 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:25.418795 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:25.417266 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 01:27:25.418795 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:25.417324 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" podUID="f79971c3-5d7b-4fd1-a84f-46b2c0190af6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 01:27:25.422170 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:25.422066 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:25.576147 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:25.576110 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:25.710351 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:25.710319 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:27:25.711155 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:25.711128 5300 patch_prober.go:29] interesting pod/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 01:27:25.711326 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:25.711301 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" podUID="85d29953-787a-49a2-87d5-1514f1c425ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 01:27:25.908523 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:25.907745 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43d688cb-3c2f-4d2d-84d6-1dbadcdb2985" path="/var/lib/kubelet/pods/43d688cb-3c2f-4d2d-84d6-1dbadcdb2985/volumes" Oct 30 01:27:25.997938 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:25.997826 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:26.402788 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:26.401641 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 01:27:26.402788 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:26.401697 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" podUID="f79971c3-5d7b-4fd1-a84f-46b2c0190af6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 01:27:26.700551 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:26.700477 5300 patch_prober.go:29] interesting pod/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 01:27:26.700823 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:26.700792 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" podUID="85d29953-787a-49a2-87d5-1514f1c425ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 01:27:27.402003 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:27.401972 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 01:27:27.402449 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:27.402030 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" podUID="f79971c3-5d7b-4fd1-a84f-46b2c0190af6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 01:27:27.487788 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:27.486275 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:27.865990 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:27.849738 5300 patch_prober.go:29] interesting pod/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 01:27:27.865990 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:27.849919 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" podUID="85d29953-787a-49a2-87d5-1514f1c425ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 01:27:27.915057 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:27.912109 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:28.044796 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:28.025865 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc"] Oct 30 01:27:28.074244 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:28.074213 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc"] Oct 30 01:27:28.074479 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:28.074367 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc" Oct 30 01:27:28.108036 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:28.107989 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-eaf753-dockercfg-bcbtc\"" Oct 30 01:27:28.108210 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:28.108009 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-eaf753\"" Oct 30 01:27:28.112199 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:28.112170 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db"] Oct 30 01:27:28.193317 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:28.193222 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26bb7124-a395-4690-a7dc-9d2236e9a083-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc\" (UID: \"26bb7124-a395-4690-a7dc-9d2236e9a083\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc" Oct 30 01:27:28.193317 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:28.193309 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lfh9\" (UniqueName: \"kubernetes.io/projected/26bb7124-a395-4690-a7dc-9d2236e9a083-kube-api-access-6lfh9\") pod \"payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc\" (UID: \"26bb7124-a395-4690-a7dc-9d2236e9a083\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc" Oct 30 01:27:28.297569 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:28.296303 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:28.299158 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:28.299120 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26bb7124-a395-4690-a7dc-9d2236e9a083-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc\" (UID: \"26bb7124-a395-4690-a7dc-9d2236e9a083\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc" Oct 30 01:27:28.299321 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:28.299206 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lfh9\" (UniqueName: \"kubernetes.io/projected/26bb7124-a395-4690-a7dc-9d2236e9a083-kube-api-access-6lfh9\") pod \"payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc\" (UID: \"26bb7124-a395-4690-a7dc-9d2236e9a083\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc" Oct 30 01:27:28.301916 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:28.300223 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26bb7124-a395-4690-a7dc-9d2236e9a083-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc\" (UID: \"26bb7124-a395-4690-a7dc-9d2236e9a083\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc" Oct 30 01:27:28.390800 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:28.390096 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lfh9\" (UniqueName: \"kubernetes.io/projected/26bb7124-a395-4690-a7dc-9d2236e9a083-kube-api-access-6lfh9\") pod \"payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc\" (UID: \"26bb7124-a395-4690-a7dc-9d2236e9a083\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc" Oct 30 01:27:28.390800 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:28.390497 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:28.400591 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:28.400561 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc" Oct 30 01:27:28.406158 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:28.404677 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 01:27:28.406158 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:28.404740 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" podUID="f79971c3-5d7b-4fd1-a84f-46b2c0190af6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 01:27:28.701322 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:28.700582 5300 patch_prober.go:29] interesting pod/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 01:27:28.701322 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:28.700668 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" podUID="85d29953-787a-49a2-87d5-1514f1c425ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 01:27:29.038612 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:29.038031 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc"] Oct 30 01:27:29.401140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:29.401106 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 01:27:29.401288 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:29.401154 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" podUID="f79971c3-5d7b-4fd1-a84f-46b2c0190af6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 01:27:29.700034 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:29.699928 5300 patch_prober.go:29] interesting pod/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 01:27:29.700034 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:29.699989 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" podUID="85d29953-787a-49a2-87d5-1514f1c425ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 01:27:29.999086 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:29.999053 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:30.127791 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:30.127694 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:30.149184 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:30.149140 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc" event={"ID":"26bb7124-a395-4690-a7dc-9d2236e9a083","Type":"ContainerStarted","Data":"f2a7b43c216409deae6851451548d7cfbca94c71979098835da05f49a4d859d1"} Oct 30 01:27:30.176356 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:30.176327 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc" Oct 30 01:27:30.176598 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:30.176577 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc" event={"ID":"26bb7124-a395-4690-a7dc-9d2236e9a083","Type":"ContainerStarted","Data":"9cc0d93aafaf3305ed85d02fc531292300b870d51b4909e3e915b523ad868a36"} Oct 30 01:27:30.190055 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:30.190015 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb" podUID="aced2f61-18a5-4919-88a5-a76f6952085d" containerName="payload-pre-processing" containerID="cri-o://c23b023458ccdad8d53dc0086f79abb6aa5407fefcb17c7ca89ac4ab639663e2" gracePeriod=25 Oct 30 01:27:30.401383 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:30.401355 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body= Oct 30 01:27:30.401644 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:30.401592 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" podUID="f79971c3-5d7b-4fd1-a84f-46b2c0190af6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" Oct 30 01:27:30.698469 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:30.698429 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb" Oct 30 01:27:30.700371 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:30.700344 5300 patch_prober.go:29] interesting pod/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 01:27:30.700648 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:30.700403 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" podUID="85d29953-787a-49a2-87d5-1514f1c425ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 01:27:30.706252 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:30.706218 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aced2f61-18a5-4919-88a5-a76f6952085d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aced2f61-18a5-4919-88a5-a76f6952085d-plugins-config-volume\") pod \"aced2f61-18a5-4919-88a5-a76f6952085d\" (UID: \"aced2f61-18a5-4919-88a5-a76f6952085d\") " Oct 30 01:27:30.706554 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:30.706534 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aced2f61-18a5-4919-88a5-a76f6952085d-plugins-config-volume" pod "aced2f61-18a5-4919-88a5-a76f6952085d" (UID: "aced2f61-18a5-4919-88a5-a76f6952085d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:30.707150 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:30.707131 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aced2f61-18a5-4919-88a5-a76f6952085d-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:27:30.752147 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:30.752083 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc" podStartSLOduration=3.752062793 podStartE2EDuration="3.752062793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:30.264397958 +0000 UTC m=+2650.580266284" watchObservedRunningTime="2026-10-30 01:27:30.752062793 +0000 UTC m=+2651.067931122" Oct 30 01:27:30.808325 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:30.808288 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aced2f61-18a5-4919-88a5-a76f6952085d-kube-api-access-dk4vl\" (UniqueName: \"kubernetes.io/projected/aced2f61-18a5-4919-88a5-a76f6952085d-kube-api-access-dk4vl\") pod \"aced2f61-18a5-4919-88a5-a76f6952085d\" (UID: \"aced2f61-18a5-4919-88a5-a76f6952085d\") " Oct 30 01:27:30.810692 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:30.810654 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aced2f61-18a5-4919-88a5-a76f6952085d-kube-api-access-dk4vl" pod "aced2f61-18a5-4919-88a5-a76f6952085d" (UID: "aced2f61-18a5-4919-88a5-a76f6952085d"). InnerVolumeSpecName "kube-api-access-dk4vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:30.910814 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:30.910681 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk4vl\" (UniqueName: \"kubernetes.io/projected/aced2f61-18a5-4919-88a5-a76f6952085d-kube-api-access-dk4vl\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:27:31.042336 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:31.042300 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:31.160794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:31.160726 5300 generic.go:356] "Generic (PLEG): container finished" podID="aced2f61-18a5-4919-88a5-a76f6952085d" containerID="c23b023458ccdad8d53dc0086f79abb6aa5407fefcb17c7ca89ac4ab639663e2" exitCode=0 Oct 30 01:27:31.160794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:31.160792 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb" Oct 30 01:27:31.161033 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:31.160820 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb" event={"ID":"aced2f61-18a5-4919-88a5-a76f6952085d","Type":"ContainerDied","Data":"c23b023458ccdad8d53dc0086f79abb6aa5407fefcb17c7ca89ac4ab639663e2"} Oct 30 01:27:31.161033 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:31.160878 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb" event={"ID":"aced2f61-18a5-4919-88a5-a76f6952085d","Type":"ContainerDied","Data":"2590ee02b5c229a7a0eac913ed2b4117086d00aa1aafb44629b91adf5d204930"} Oct 30 01:27:31.161033 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:31.160905 5300 scope.go:122] "RemoveContainer" containerID="c23b023458ccdad8d53dc0086f79abb6aa5407fefcb17c7ca89ac4ab639663e2" Oct 30 01:27:31.174090 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:31.174064 5300 scope.go:122] "RemoveContainer" containerID="c23b023458ccdad8d53dc0086f79abb6aa5407fefcb17c7ca89ac4ab639663e2" Oct 30 01:27:31.174438 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:27:31.174408 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c23b023458ccdad8d53dc0086f79abb6aa5407fefcb17c7ca89ac4ab639663e2\": container with ID starting with c23b023458ccdad8d53dc0086f79abb6aa5407fefcb17c7ca89ac4ab639663e2 not found: ID does not exist" containerID="c23b023458ccdad8d53dc0086f79abb6aa5407fefcb17c7ca89ac4ab639663e2" Oct 30 01:27:31.174538 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:31.174453 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c23b023458ccdad8d53dc0086f79abb6aa5407fefcb17c7ca89ac4ab639663e2"} err="failed to get container status \"c23b023458ccdad8d53dc0086f79abb6aa5407fefcb17c7ca89ac4ab639663e2\": rpc error: code = NotFound desc = could not find container \"c23b023458ccdad8d53dc0086f79abb6aa5407fefcb17c7ca89ac4ab639663e2\": container with ID starting with c23b023458ccdad8d53dc0086f79abb6aa5407fefcb17c7ca89ac4ab639663e2 not found: ID does not exist" Oct 30 01:27:31.223422 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:31.223385 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb"] Oct 30 01:27:31.232619 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:31.232591 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73831d-58cbc7867f-tcrmb"] Oct 30 01:27:31.294943 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:31.294912 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:31.407008 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:31.406980 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:31.407460 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:31.407437 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:31.408155 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:31.408136 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:27:31.705219 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:31.705187 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:27:31.705676 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:31.705525 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:27:31.706253 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:31.706229 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:27:31.873602 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:31.873567 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aced2f61-18a5-4919-88a5-a76f6952085d" path="/var/lib/kubelet/pods/aced2f61-18a5-4919-88a5-a76f6952085d/volumes" Oct 30 01:27:32.243707 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:32.243678 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:33.113519 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:33.113437 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db" podUID="53efc96a-851b-49fd-afee-b158c3a849b6" containerName="payload-processing" containerID="cri-o://d19cc08c3def581d9ef767b88d9ac1ca660b4788afd09ead4d0d5f5573d25eea" gracePeriod=25 Oct 30 01:27:34.057432 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:34.057402 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:35.048100 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:35.048063 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:36.427229 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:36.427195 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:37.879971 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:37.879935 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db" podUID="53efc96a-851b-49fd-afee-b158c3a849b6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:27:38.137662 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:38.137574 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:39.859387 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:39.859287 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:40.467817 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:40.467781 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:41.164230 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:41.164203 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc" Oct 30 01:27:41.545367 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:41.545332 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:42.864457 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:42.864406 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:42.962716 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:42.962678 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:43.316914 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:43.316880 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:44.923047 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:44.923010 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:46.625082 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:46.625045 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:47.879504 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:47.879467 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db" podUID="53efc96a-851b-49fd-afee-b158c3a849b6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:27:47.902042 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:47.902014 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:48.335981 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:48.335922 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:48.445563 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:48.445522 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:48.541491 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:48.541457 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:49.038978 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:49.038947 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:50.156220 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:50.156186 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:51.595146 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:27:51.595109 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:27:51.776012 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:51.775979 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:53.070714 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:53.070684 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz"] Oct 30 01:27:53.071192 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:53.070886 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" podUID="c8749ea6-d001-45fd-b7bd-956be5364f8c" containerName="istio-proxy" containerID="cri-o://8c21f52f41747dc8404c42511b14dc856a04a11f9da3129fcd6522253e2f8e1b" gracePeriod=30 Oct 30 01:27:53.390177 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:53.390082 5300 generic.go:356] "Generic (PLEG): container finished" podID="53efc96a-851b-49fd-afee-b158c3a849b6" containerID="d19cc08c3def581d9ef767b88d9ac1ca660b4788afd09ead4d0d5f5573d25eea" exitCode=0 Oct 30 01:27:53.390341 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:53.390290 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db" event={"ID":"53efc96a-851b-49fd-afee-b158c3a849b6","Type":"ContainerDied","Data":"d19cc08c3def581d9ef767b88d9ac1ca660b4788afd09ead4d0d5f5573d25eea"} Oct 30 01:27:53.416016 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:53.415987 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:53.430011 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:53.429988 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db" Oct 30 01:27:53.535417 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:53.535348 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53efc96a-851b-49fd-afee-b158c3a849b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53efc96a-851b-49fd-afee-b158c3a849b6-plugins-config-volume\") pod \"53efc96a-851b-49fd-afee-b158c3a849b6\" (UID: \"53efc96a-851b-49fd-afee-b158c3a849b6\") " Oct 30 01:27:53.535614 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:53.535521 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53efc96a-851b-49fd-afee-b158c3a849b6-kube-api-access-cg8z7\" (UniqueName: \"kubernetes.io/projected/53efc96a-851b-49fd-afee-b158c3a849b6-kube-api-access-cg8z7\") pod \"53efc96a-851b-49fd-afee-b158c3a849b6\" (UID: \"53efc96a-851b-49fd-afee-b158c3a849b6\") " Oct 30 01:27:53.536146 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:53.535752 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53efc96a-851b-49fd-afee-b158c3a849b6-plugins-config-volume" pod "53efc96a-851b-49fd-afee-b158c3a849b6" (UID: "53efc96a-851b-49fd-afee-b158c3a849b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:53.536389 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:53.536229 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53efc96a-851b-49fd-afee-b158c3a849b6-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:27:53.538895 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:53.538862 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53efc96a-851b-49fd-afee-b158c3a849b6-kube-api-access-cg8z7" pod "53efc96a-851b-49fd-afee-b158c3a849b6" (UID: "53efc96a-851b-49fd-afee-b158c3a849b6"). InnerVolumeSpecName "kube-api-access-cg8z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:53.637992 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:53.637960 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg8z7\" (UniqueName: \"kubernetes.io/projected/53efc96a-851b-49fd-afee-b158c3a849b6-kube-api-access-cg8z7\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:27:54.399894 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:54.399863 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db" Oct 30 01:27:54.400395 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:54.399888 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db" event={"ID":"53efc96a-851b-49fd-afee-b158c3a849b6","Type":"ContainerDied","Data":"94c26529e2fd1fd4bd78707a83f75f11a4ad8511edfca1e53830c17118461de2"} Oct 30 01:27:54.400395 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:54.399959 5300 scope.go:122] "RemoveContainer" containerID="d19cc08c3def581d9ef767b88d9ac1ca660b4788afd09ead4d0d5f5573d25eea" Oct 30 01:27:54.459939 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:54.459907 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db"] Oct 30 01:27:54.474016 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:54.473983 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-69dc90-6b769d56d8-v49db"] Oct 30 01:27:54.952612 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:54.952579 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:55.072116 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:55.072077 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:55.258204 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:55.258171 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:55.304946 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:55.304911 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw"] Oct 30 01:27:55.305177 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:55.305149 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" podUID="488a9458-8639-4ae8-9cd4-c839b1662099" containerName="istio-proxy" containerID="cri-o://0bd74398b7a2a101fb99c37b390218cd0a9b2ac659004c1cfd3075cd2516dd84" gracePeriod=30 Oct 30 01:27:55.359394 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:55.359363 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:55.823969 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:55.823922 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:55.824372 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:55.824245 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53efc96a-851b-49fd-afee-b158c3a849b6" path="/var/lib/kubelet/pods/53efc96a-851b-49fd-afee-b158c3a849b6/volumes" Oct 30 01:27:56.798171 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:56.798137 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:57.324201 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:57.324163 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:57.435854 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:57.430220 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:57.562182 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:57.562153 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:57.962611 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:57.962576 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:58.376515 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.376491 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:27:58.552581 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.552483 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-credential-socket\") pod \"c8749ea6-d001-45fd-b7bd-956be5364f8c\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " Oct 30 01:27:58.552581 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.552539 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-data\") pod \"c8749ea6-d001-45fd-b7bd-956be5364f8c\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " Oct 30 01:27:58.552889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.552740 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-token\" (UniqueName: \"kubernetes.io/projected/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-token\") pod \"c8749ea6-d001-45fd-b7bd-956be5364f8c\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " Oct 30 01:27:58.552889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.552807 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8749ea6-d001-45fd-b7bd-956be5364f8c-kube-api-access-wpjfl\" (UniqueName: \"kubernetes.io/projected/c8749ea6-d001-45fd-b7bd-956be5364f8c-kube-api-access-wpjfl\") pod \"c8749ea6-d001-45fd-b7bd-956be5364f8c\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " Oct 30 01:27:58.553015 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.552931 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-envoy\") pod \"c8749ea6-d001-45fd-b7bd-956be5364f8c\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " Oct 30 01:27:58.553015 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.552993 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-workload-certs\") pod \"c8749ea6-d001-45fd-b7bd-956be5364f8c\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " Oct 30 01:27:58.553131 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.553111 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8749ea6-d001-45fd-b7bd-956be5364f8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8749ea6-d001-45fd-b7bd-956be5364f8c-istiod-ca-cert\") pod \"c8749ea6-d001-45fd-b7bd-956be5364f8c\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " Oct 30 01:27:58.553310 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.553217 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-podinfo\") pod \"c8749ea6-d001-45fd-b7bd-956be5364f8c\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " Oct 30 01:27:58.553310 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.553264 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-workload-socket\") pod \"c8749ea6-d001-45fd-b7bd-956be5364f8c\" (UID: \"c8749ea6-d001-45fd-b7bd-956be5364f8c\") " Oct 30 01:27:58.553920 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.553350 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-data" pod "c8749ea6-d001-45fd-b7bd-956be5364f8c" (UID: "c8749ea6-d001-45fd-b7bd-956be5364f8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:58.553920 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.553654 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-workload-certs" pod "c8749ea6-d001-45fd-b7bd-956be5364f8c" (UID: "c8749ea6-d001-45fd-b7bd-956be5364f8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:58.553920 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.553908 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-workload-socket" pod "c8749ea6-d001-45fd-b7bd-956be5364f8c" (UID: "c8749ea6-d001-45fd-b7bd-956be5364f8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:58.554111 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.553928 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8749ea6-d001-45fd-b7bd-956be5364f8c-istiod-ca-cert" pod "c8749ea6-d001-45fd-b7bd-956be5364f8c" (UID: "c8749ea6-d001-45fd-b7bd-956be5364f8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:58.554295 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.554270 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8749ea6-d001-45fd-b7bd-956be5364f8c-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:27:58.554374 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.554298 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:27:58.554374 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.554313 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:27:58.554374 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.554327 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:27:58.554621 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.554570 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-credential-socket" pod "c8749ea6-d001-45fd-b7bd-956be5364f8c" (UID: "c8749ea6-d001-45fd-b7bd-956be5364f8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:58.561146 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.556172 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8749ea6-d001-45fd-b7bd-956be5364f8c-kube-api-access-wpjfl" pod "c8749ea6-d001-45fd-b7bd-956be5364f8c" (UID: "c8749ea6-d001-45fd-b7bd-956be5364f8c"). InnerVolumeSpecName "kube-api-access-wpjfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:58.561146 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.556659 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-envoy" pod "c8749ea6-d001-45fd-b7bd-956be5364f8c" (UID: "c8749ea6-d001-45fd-b7bd-956be5364f8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:27:58.561146 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.557489 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-token" pod "c8749ea6-d001-45fd-b7bd-956be5364f8c" (UID: "c8749ea6-d001-45fd-b7bd-956be5364f8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:58.561146 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.558716 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-podinfo" pod "c8749ea6-d001-45fd-b7bd-956be5364f8c" (UID: "c8749ea6-d001-45fd-b7bd-956be5364f8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:27:58.561146 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.560138 5300 generic.go:356] "Generic (PLEG): container finished" podID="c8749ea6-d001-45fd-b7bd-956be5364f8c" containerID="8c21f52f41747dc8404c42511b14dc856a04a11f9da3129fcd6522253e2f8e1b" exitCode=0 Oct 30 01:27:58.561146 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.560197 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" Oct 30 01:27:58.561146 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.560223 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" event={"ID":"c8749ea6-d001-45fd-b7bd-956be5364f8c","Type":"ContainerDied","Data":"8c21f52f41747dc8404c42511b14dc856a04a11f9da3129fcd6522253e2f8e1b"} Oct 30 01:27:58.561146 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.560289 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz" event={"ID":"c8749ea6-d001-45fd-b7bd-956be5364f8c","Type":"ContainerDied","Data":"d0268dfa43c4eb3942f5f6b7724165408cc60c2df6fc00c0dcb07d2cf2310072"} Oct 30 01:27:58.561146 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.560318 5300 scope.go:122] "RemoveContainer" containerID="8c21f52f41747dc8404c42511b14dc856a04a11f9da3129fcd6522253e2f8e1b" Oct 30 01:27:58.592289 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.592270 5300 scope.go:122] "RemoveContainer" containerID="8c21f52f41747dc8404c42511b14dc856a04a11f9da3129fcd6522253e2f8e1b" Oct 30 01:27:58.592572 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:27:58.592550 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c21f52f41747dc8404c42511b14dc856a04a11f9da3129fcd6522253e2f8e1b\": container with ID starting with 8c21f52f41747dc8404c42511b14dc856a04a11f9da3129fcd6522253e2f8e1b not found: ID does not exist" containerID="8c21f52f41747dc8404c42511b14dc856a04a11f9da3129fcd6522253e2f8e1b" Oct 30 01:27:58.592640 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.592584 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c21f52f41747dc8404c42511b14dc856a04a11f9da3129fcd6522253e2f8e1b"} err="failed to get container status \"8c21f52f41747dc8404c42511b14dc856a04a11f9da3129fcd6522253e2f8e1b\": rpc error: code = NotFound desc = could not find container \"8c21f52f41747dc8404c42511b14dc856a04a11f9da3129fcd6522253e2f8e1b\": container with ID starting with 8c21f52f41747dc8404c42511b14dc856a04a11f9da3129fcd6522253e2f8e1b not found: ID does not exist" Oct 30 01:27:58.630070 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.630040 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz"] Oct 30 01:27:58.633895 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.633870 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-29145c-openshift-default-56b4496f7f-qpxdz"] Oct 30 01:27:58.655467 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.655440 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:27:58.655467 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.655465 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpjfl\" (UniqueName: \"kubernetes.io/projected/c8749ea6-d001-45fd-b7bd-956be5364f8c-kube-api-access-wpjfl\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:27:58.655616 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.655479 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:27:58.655616 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.655491 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8749ea6-d001-45fd-b7bd-956be5364f8c-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:27:58.655616 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.655501 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8749ea6-d001-45fd-b7bd-956be5364f8c-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:27:58.883277 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:58.883178 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:59.031652 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:59.031616 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:59.711626 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:59.711574 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:27:59.834382 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:27:59.834347 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8749ea6-d001-45fd-b7bd-956be5364f8c" path="/var/lib/kubelet/pods/c8749ea6-d001-45fd-b7bd-956be5364f8c/volumes" Oct 30 01:28:00.325682 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.325645 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:00.435373 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.435329 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:00.540834 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.540646 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:00.586867 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.586537 5300 generic.go:356] "Generic (PLEG): container finished" podID="488a9458-8639-4ae8-9cd4-c839b1662099" containerID="0bd74398b7a2a101fb99c37b390218cd0a9b2ac659004c1cfd3075cd2516dd84" exitCode=0 Oct 30 01:28:00.586867 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.586652 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" event={"ID":"488a9458-8639-4ae8-9cd4-c839b1662099","Type":"ContainerDied","Data":"0bd74398b7a2a101fb99c37b390218cd0a9b2ac659004c1cfd3075cd2516dd84"} Oct 30 01:28:00.669301 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.669277 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:28:00.759793 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.756467 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/488a9458-8639-4ae8-9cd4-c839b1662099-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/488a9458-8639-4ae8-9cd4-c839b1662099-istio-podinfo\") pod \"488a9458-8639-4ae8-9cd4-c839b1662099\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " Oct 30 01:28:00.759793 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.756930 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-credential-socket\") pod \"488a9458-8639-4ae8-9cd4-c839b1662099\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " Oct 30 01:28:00.759793 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.756977 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-istio-envoy\") pod \"488a9458-8639-4ae8-9cd4-c839b1662099\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " Oct 30 01:28:00.759793 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.757040 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-workload-certs\") pod \"488a9458-8639-4ae8-9cd4-c839b1662099\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " Oct 30 01:28:00.759793 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.757088 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/488a9458-8639-4ae8-9cd4-c839b1662099-kube-api-access-fzmgh\" (UniqueName: \"kubernetes.io/projected/488a9458-8639-4ae8-9cd4-c839b1662099-kube-api-access-fzmgh\") pod \"488a9458-8639-4ae8-9cd4-c839b1662099\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " Oct 30 01:28:00.759793 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.757113 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-workload-socket\") pod \"488a9458-8639-4ae8-9cd4-c839b1662099\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " Oct 30 01:28:00.759793 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.757171 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-istio-data\") pod \"488a9458-8639-4ae8-9cd4-c839b1662099\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " Oct 30 01:28:00.759793 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.757248 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/488a9458-8639-4ae8-9cd4-c839b1662099-istio-token\" (UniqueName: \"kubernetes.io/projected/488a9458-8639-4ae8-9cd4-c839b1662099-istio-token\") pod \"488a9458-8639-4ae8-9cd4-c839b1662099\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " Oct 30 01:28:00.759793 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.757846 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-workload-socket" pod "488a9458-8639-4ae8-9cd4-c839b1662099" (UID: "488a9458-8639-4ae8-9cd4-c839b1662099"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:00.759793 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.757930 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/488a9458-8639-4ae8-9cd4-c839b1662099-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/488a9458-8639-4ae8-9cd4-c839b1662099-istiod-ca-cert\") pod \"488a9458-8639-4ae8-9cd4-c839b1662099\" (UID: \"488a9458-8639-4ae8-9cd4-c839b1662099\") " Oct 30 01:28:00.759793 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.758997 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.773512 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.761500 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/488a9458-8639-4ae8-9cd4-c839b1662099-istiod-ca-cert" pod "488a9458-8639-4ae8-9cd4-c839b1662099" (UID: "488a9458-8639-4ae8-9cd4-c839b1662099"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:00.773512 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.761740 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-istio-data" pod "488a9458-8639-4ae8-9cd4-c839b1662099" (UID: "488a9458-8639-4ae8-9cd4-c839b1662099"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:00.773512 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.761834 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-credential-socket" pod "488a9458-8639-4ae8-9cd4-c839b1662099" (UID: "488a9458-8639-4ae8-9cd4-c839b1662099"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:00.773512 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.761935 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-workload-certs" pod "488a9458-8639-4ae8-9cd4-c839b1662099" (UID: "488a9458-8639-4ae8-9cd4-c839b1662099"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:00.773512 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.772305 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488a9458-8639-4ae8-9cd4-c839b1662099-istio-token" pod "488a9458-8639-4ae8-9cd4-c839b1662099" (UID: "488a9458-8639-4ae8-9cd4-c839b1662099"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:00.774173 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.774144 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/488a9458-8639-4ae8-9cd4-c839b1662099-istio-podinfo" pod "488a9458-8639-4ae8-9cd4-c839b1662099" (UID: "488a9458-8639-4ae8-9cd4-c839b1662099"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:28:00.778248 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.776407 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-istio-envoy" pod "488a9458-8639-4ae8-9cd4-c839b1662099" (UID: "488a9458-8639-4ae8-9cd4-c839b1662099"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:00.778248 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.777505 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488a9458-8639-4ae8-9cd4-c839b1662099-kube-api-access-fzmgh" pod "488a9458-8639-4ae8-9cd4-c839b1662099" (UID: "488a9458-8639-4ae8-9cd4-c839b1662099"). InnerVolumeSpecName "kube-api-access-fzmgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:00.862992 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.862896 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.862992 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.862929 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.862992 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.862945 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.862992 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.862958 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzmgh\" (UniqueName: \"kubernetes.io/projected/488a9458-8639-4ae8-9cd4-c839b1662099-kube-api-access-fzmgh\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.862992 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.862975 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/488a9458-8639-4ae8-9cd4-c839b1662099-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.862992 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.862988 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/488a9458-8639-4ae8-9cd4-c839b1662099-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.863381 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.863001 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/488a9458-8639-4ae8-9cd4-c839b1662099-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.863381 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.863016 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/488a9458-8639-4ae8-9cd4-c839b1662099-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.911115 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:00.911077 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:01.038926 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:01.038889 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:01.574931 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:01.574898 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:01.597353 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:01.597325 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" Oct 30 01:28:01.597539 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:01.597322 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw" event={"ID":"488a9458-8639-4ae8-9cd4-c839b1662099","Type":"ContainerDied","Data":"5fff944e6f27a87f407b110e50e872d93f28fe763952ab32dd21ff7d6c141ef2"} Oct 30 01:28:01.597539 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:01.597466 5300 scope.go:122] "RemoveContainer" containerID="0bd74398b7a2a101fb99c37b390218cd0a9b2ac659004c1cfd3075cd2516dd84" Oct 30 01:28:01.670215 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:01.670180 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw"] Oct 30 01:28:01.678100 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:01.678063 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:01.684333 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:01.684305 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-69dc90-openshift-default-df55b565f-cfbxw"] Oct 30 01:28:01.831401 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:01.831315 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="488a9458-8639-4ae8-9cd4-c839b1662099" path="/var/lib/kubelet/pods/488a9458-8639-4ae8-9cd4-c839b1662099/volumes" Oct 30 01:28:02.510821 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:02.510785 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:04.313945 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:04.313912 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:04.427135 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:04.427102 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:04.646903 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:04.646818 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:04.751128 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:04.751094 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:05.250090 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:05.249342 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:05.952887 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:05.952854 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:07.150253 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:07.150202 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:07.267561 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:07.267526 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:07.494699 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:07.494667 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:07.625159 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:07.625130 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:07.866432 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:07.866358 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:08.131276 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:08.131182 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:08.449908 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:08.449809 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:08.551176 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:08.551147 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:08.651570 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:08.651529 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:08.757430 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:08.757393 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:08.962154 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:08.962107 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:10.534026 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:10.533980 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:11.874006 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:11.873972 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:12.001564 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:12.001532 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:12.230013 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:12.229979 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:12.410685 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:12.410647 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:14.131533 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:14.131492 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:15.457480 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:15.457429 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:15.556275 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:15.556244 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:15.841062 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:15.840637 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:16.187427 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:16.187345 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:16.996540 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:16.996504 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:17.121565 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:17.121535 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:18.099187 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:18.099155 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:20.101705 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:20.101667 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk"] Oct 30 01:28:20.199171 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:20.199133 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:20.341795 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:20.322084 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:20.582247 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:20.582218 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:20.698334 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:20.698299 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:20.963355 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:20.963271 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq"] Oct 30 01:28:21.125035 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:21.125004 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:21.229715 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:21.229618 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:21.429148 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:21.429114 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:21.573753 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:21.573671 5300 scope.go:122] "RemoveContainer" containerID="31d6f878038a28754160021e18135b16ca02d2f109676353bbe5748f80cd8227" Oct 30 01:28:21.597657 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:28:21.597619 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:28:22.052785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:22.050730 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:22.502064 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:22.502030 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:22.602938 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:22.602909 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:22.723021 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:22.722981 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:23.160046 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:23.159989 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:23.425137 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:23.425053 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:24.472253 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:24.472216 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:25.102345 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:25.102304 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk" podUID="6b19732a-da65-4b29-a2e7-a2017ce9b815" containerName="payload-pre-processing" containerID="cri-o://5c00e6157e657fe49107d18a97654b150248268ce3896e00e5d8b3ea63cde9cf" gracePeriod=25 Oct 30 01:28:25.963814 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:25.963734 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq" podUID="3f305e4e-92ce-4b37-adb8-3a8106e3854e" containerName="payload-processing" containerID="cri-o://7820ee8e3de2da43875efe602c7eeec4007d333a319ef415658b27eb149e4393" gracePeriod=25 Oct 30 01:28:26.050103 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:26.050068 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:26.290279 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:26.290253 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq" Oct 30 01:28:26.325395 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:26.325364 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f305e4e-92ce-4b37-adb8-3a8106e3854e-kube-api-access-6w4nj\" (UniqueName: \"kubernetes.io/projected/3f305e4e-92ce-4b37-adb8-3a8106e3854e-kube-api-access-6w4nj\") pod \"3f305e4e-92ce-4b37-adb8-3a8106e3854e\" (UID: \"3f305e4e-92ce-4b37-adb8-3a8106e3854e\") " Oct 30 01:28:26.325549 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:26.325530 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f305e4e-92ce-4b37-adb8-3a8106e3854e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f305e4e-92ce-4b37-adb8-3a8106e3854e-plugins-config-volume\") pod \"3f305e4e-92ce-4b37-adb8-3a8106e3854e\" (UID: \"3f305e4e-92ce-4b37-adb8-3a8106e3854e\") " Oct 30 01:28:26.325897 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:26.325866 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f305e4e-92ce-4b37-adb8-3a8106e3854e-plugins-config-volume" pod "3f305e4e-92ce-4b37-adb8-3a8106e3854e" (UID: "3f305e4e-92ce-4b37-adb8-3a8106e3854e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:26.326187 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:26.326158 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f305e4e-92ce-4b37-adb8-3a8106e3854e-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:26.327814 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:26.327785 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f305e4e-92ce-4b37-adb8-3a8106e3854e-kube-api-access-6w4nj" pod "3f305e4e-92ce-4b37-adb8-3a8106e3854e" (UID: "3f305e4e-92ce-4b37-adb8-3a8106e3854e"). InnerVolumeSpecName "kube-api-access-6w4nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:26.428083 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:26.428052 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w4nj\" (UniqueName: \"kubernetes.io/projected/3f305e4e-92ce-4b37-adb8-3a8106e3854e-kube-api-access-6w4nj\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:26.719423 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:26.719320 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:26.839170 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:26.839141 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:26.960426 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:26.960393 5300 generic.go:356] "Generic (PLEG): container finished" podID="3f305e4e-92ce-4b37-adb8-3a8106e3854e" containerID="7820ee8e3de2da43875efe602c7eeec4007d333a319ef415658b27eb149e4393" exitCode=0 Oct 30 01:28:26.960605 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:26.960438 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq" Oct 30 01:28:26.960605 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:26.960477 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq" event={"ID":"3f305e4e-92ce-4b37-adb8-3a8106e3854e","Type":"ContainerDied","Data":"7820ee8e3de2da43875efe602c7eeec4007d333a319ef415658b27eb149e4393"} Oct 30 01:28:26.960605 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:26.960532 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq" event={"ID":"3f305e4e-92ce-4b37-adb8-3a8106e3854e","Type":"ContainerDied","Data":"5b560ed9cba13351b5fe2ba765eee1f3494f8e08b896dcc1917cf7e5bb9e09ea"} Oct 30 01:28:26.960605 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:26.960554 5300 scope.go:122] "RemoveContainer" containerID="7820ee8e3de2da43875efe602c7eeec4007d333a319ef415658b27eb149e4393" Oct 30 01:28:26.971482 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:26.971466 5300 scope.go:122] "RemoveContainer" containerID="7820ee8e3de2da43875efe602c7eeec4007d333a319ef415658b27eb149e4393" Oct 30 01:28:26.971795 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:28:26.971737 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7820ee8e3de2da43875efe602c7eeec4007d333a319ef415658b27eb149e4393\": container with ID starting with 7820ee8e3de2da43875efe602c7eeec4007d333a319ef415658b27eb149e4393 not found: ID does not exist" containerID="7820ee8e3de2da43875efe602c7eeec4007d333a319ef415658b27eb149e4393" Oct 30 01:28:26.971795 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:26.971774 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7820ee8e3de2da43875efe602c7eeec4007d333a319ef415658b27eb149e4393"} err="failed to get container status \"7820ee8e3de2da43875efe602c7eeec4007d333a319ef415658b27eb149e4393\": rpc error: code = NotFound desc = could not find container \"7820ee8e3de2da43875efe602c7eeec4007d333a319ef415658b27eb149e4393\": container with ID starting with 7820ee8e3de2da43875efe602c7eeec4007d333a319ef415658b27eb149e4393 not found: ID does not exist" Oct 30 01:28:27.020720 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:27.020693 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq"] Oct 30 01:28:27.034385 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:27.034363 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-29bb56-57fb68ccf6-vpkgq"] Oct 30 01:28:27.824197 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:27.824165 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f305e4e-92ce-4b37-adb8-3a8106e3854e" path="/var/lib/kubelet/pods/3f305e4e-92ce-4b37-adb8-3a8106e3854e/volumes" Oct 30 01:28:28.124488 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:28.124370 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:28.227288 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:28.227248 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:28.998776 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:28.998718 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk" podUID="6b19732a-da65-4b29-a2e7-a2017ce9b815" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:28:29.970865 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:29.970526 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:30.082964 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:30.082926 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:30.189329 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:30.189295 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:30.572828 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:30.571368 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:30.675495 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:30.675458 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:30.849387 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:30.849049 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5"] Oct 30 01:28:30.849860 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:30.849824 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" podUID="cfe18c62-795e-43b0-826c-35b3e91d9890" containerName="istio-proxy" containerID="cri-o://93c02683d1aa5fc4039b3aa66b43c7e274645fa4ca61129446b39492efdba20c" gracePeriod=30 Oct 30 01:28:30.857028 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:30.857001 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:32.146646 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:32.146610 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:33.812638 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:33.812601 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:35.209820 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:35.209783 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:35.535156 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:35.535126 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:36.059623 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.059593 5300 generic.go:356] "Generic (PLEG): container finished" podID="cfe18c62-795e-43b0-826c-35b3e91d9890" containerID="93c02683d1aa5fc4039b3aa66b43c7e274645fa4ca61129446b39492efdba20c" exitCode=0 Oct 30 01:28:36.059836 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.059718 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" event={"ID":"cfe18c62-795e-43b0-826c-35b3e91d9890","Type":"ContainerDied","Data":"93c02683d1aa5fc4039b3aa66b43c7e274645fa4ca61129446b39492efdba20c"} Oct 30 01:28:36.150284 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.150260 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:28:36.247076 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.247046 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-data\") pod \"cfe18c62-795e-43b0-826c-35b3e91d9890\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " Oct 30 01:28:36.247537 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.247088 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-workload-certs\") pod \"cfe18c62-795e-43b0-826c-35b3e91d9890\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " Oct 30 01:28:36.247537 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.247183 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfe18c62-795e-43b0-826c-35b3e91d9890-kube-api-access-xsv6t\" (UniqueName: \"kubernetes.io/projected/cfe18c62-795e-43b0-826c-35b3e91d9890-kube-api-access-xsv6t\") pod \"cfe18c62-795e-43b0-826c-35b3e91d9890\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " Oct 30 01:28:36.247537 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.247310 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-envoy\") pod \"cfe18c62-795e-43b0-826c-35b3e91d9890\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " Oct 30 01:28:36.247537 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.247344 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-data" pod "cfe18c62-795e-43b0-826c-35b3e91d9890" (UID: "cfe18c62-795e-43b0-826c-35b3e91d9890"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:36.247537 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.247368 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-podinfo\") pod \"cfe18c62-795e-43b0-826c-35b3e91d9890\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " Oct 30 01:28:36.247537 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.247459 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-credential-socket\") pod \"cfe18c62-795e-43b0-826c-35b3e91d9890\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " Oct 30 01:28:36.247537 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.247464 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-workload-certs" pod "cfe18c62-795e-43b0-826c-35b3e91d9890" (UID: "cfe18c62-795e-43b0-826c-35b3e91d9890"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:36.247537 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.247498 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfe18c62-795e-43b0-826c-35b3e91d9890-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe18c62-795e-43b0-826c-35b3e91d9890-istiod-ca-cert\") pod \"cfe18c62-795e-43b0-826c-35b3e91d9890\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " Oct 30 01:28:36.248011 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.247607 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-token\" (UniqueName: \"kubernetes.io/projected/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-token\") pod \"cfe18c62-795e-43b0-826c-35b3e91d9890\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " Oct 30 01:28:36.248011 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.247709 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-workload-socket\") pod \"cfe18c62-795e-43b0-826c-35b3e91d9890\" (UID: \"cfe18c62-795e-43b0-826c-35b3e91d9890\") " Oct 30 01:28:36.248011 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.247733 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-credential-socket" pod "cfe18c62-795e-43b0-826c-35b3e91d9890" (UID: "cfe18c62-795e-43b0-826c-35b3e91d9890"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:36.248011 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.247986 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-workload-socket" pod "cfe18c62-795e-43b0-826c-35b3e91d9890" (UID: "cfe18c62-795e-43b0-826c-35b3e91d9890"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:36.248224 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.248009 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe18c62-795e-43b0-826c-35b3e91d9890-istiod-ca-cert" pod "cfe18c62-795e-43b0-826c-35b3e91d9890" (UID: "cfe18c62-795e-43b0-826c-35b3e91d9890"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:36.248579 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.248551 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:36.248681 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.248584 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:36.248681 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.248600 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:36.248681 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.248614 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:36.248681 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.248629 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe18c62-795e-43b0-826c-35b3e91d9890-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:36.249942 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.249911 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe18c62-795e-43b0-826c-35b3e91d9890-kube-api-access-xsv6t" pod "cfe18c62-795e-43b0-826c-35b3e91d9890" (UID: "cfe18c62-795e-43b0-826c-35b3e91d9890"). InnerVolumeSpecName "kube-api-access-xsv6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:36.250039 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.250015 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-token" pod "cfe18c62-795e-43b0-826c-35b3e91d9890" (UID: "cfe18c62-795e-43b0-826c-35b3e91d9890"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:36.250143 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.250126 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-envoy" pod "cfe18c62-795e-43b0-826c-35b3e91d9890" (UID: "cfe18c62-795e-43b0-826c-35b3e91d9890"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:36.250490 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.250471 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-podinfo" pod "cfe18c62-795e-43b0-826c-35b3e91d9890" (UID: "cfe18c62-795e-43b0-826c-35b3e91d9890"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:28:36.351329 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.351284 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:36.351329 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.351325 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsv6t\" (UniqueName: \"kubernetes.io/projected/cfe18c62-795e-43b0-826c-35b3e91d9890-kube-api-access-xsv6t\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:36.351559 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.351436 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:36.351559 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.351458 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe18c62-795e-43b0-826c-35b3e91d9890-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:36.969596 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:36.969551 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:37.072562 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:37.072529 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" event={"ID":"cfe18c62-795e-43b0-826c-35b3e91d9890","Type":"ContainerDied","Data":"7c1ba0d8fac3b748705ca0ac348c1af584a3a3b359b119e429fba87f66e92d04"} Oct 30 01:28:37.072562 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:37.072561 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5" Oct 30 01:28:37.072841 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:37.072597 5300 scope.go:122] "RemoveContainer" containerID="93c02683d1aa5fc4039b3aa66b43c7e274645fa4ca61129446b39492efdba20c" Oct 30 01:28:37.145553 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:37.145521 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5"] Oct 30 01:28:37.161744 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:37.161713 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-18a189-openshift-default-76698f9dc9-drtn5"] Oct 30 01:28:37.195063 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:37.195033 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:37.296269 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:37.296234 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:37.822331 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:37.822294 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfe18c62-795e-43b0-826c-35b3e91d9890" path="/var/lib/kubelet/pods/cfe18c62-795e-43b0-826c-35b3e91d9890/volumes" Oct 30 01:28:38.576155 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:38.576116 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:38.686876 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:38.686843 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:38.935078 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:38.934990 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk" podUID="6b19732a-da65-4b29-a2e7-a2017ce9b815" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:28:40.331393 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:40.331365 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:41.910919 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:41.910881 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:42.009324 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:42.009292 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:42.349499 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:42.349468 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:43.890615 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:43.890580 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:44.415327 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:44.415293 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:44.517681 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:44.517633 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:45.410119 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:45.410095 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk" Oct 30 01:28:45.517853 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:45.517820 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b19732a-da65-4b29-a2e7-a2017ce9b815-kube-api-access-lgrv5\" (UniqueName: \"kubernetes.io/projected/6b19732a-da65-4b29-a2e7-a2017ce9b815-kube-api-access-lgrv5\") pod \"6b19732a-da65-4b29-a2e7-a2017ce9b815\" (UID: \"6b19732a-da65-4b29-a2e7-a2017ce9b815\") " Oct 30 01:28:45.518121 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:45.518100 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b19732a-da65-4b29-a2e7-a2017ce9b815-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b19732a-da65-4b29-a2e7-a2017ce9b815-plugins-config-volume\") pod \"6b19732a-da65-4b29-a2e7-a2017ce9b815\" (UID: \"6b19732a-da65-4b29-a2e7-a2017ce9b815\") " Oct 30 01:28:45.518589 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:45.518561 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b19732a-da65-4b29-a2e7-a2017ce9b815-plugins-config-volume" pod "6b19732a-da65-4b29-a2e7-a2017ce9b815" (UID: "6b19732a-da65-4b29-a2e7-a2017ce9b815"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:45.520350 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:45.520320 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b19732a-da65-4b29-a2e7-a2017ce9b815-kube-api-access-lgrv5" pod "6b19732a-da65-4b29-a2e7-a2017ce9b815" (UID: "6b19732a-da65-4b29-a2e7-a2017ce9b815"). InnerVolumeSpecName "kube-api-access-lgrv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:45.597733 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:45.597702 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:45.620177 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:45.620104 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b19732a-da65-4b29-a2e7-a2017ce9b815-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:45.620177 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:45.620130 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgrv5\" (UniqueName: \"kubernetes.io/projected/6b19732a-da65-4b29-a2e7-a2017ce9b815-kube-api-access-lgrv5\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:46.139668 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:46.139633 5300 generic.go:356] "Generic (PLEG): container finished" podID="6b19732a-da65-4b29-a2e7-a2017ce9b815" containerID="5c00e6157e657fe49107d18a97654b150248268ce3896e00e5d8b3ea63cde9cf" exitCode=0 Oct 30 01:28:46.139860 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:46.139684 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk" Oct 30 01:28:46.139860 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:46.139721 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk" event={"ID":"6b19732a-da65-4b29-a2e7-a2017ce9b815","Type":"ContainerDied","Data":"5c00e6157e657fe49107d18a97654b150248268ce3896e00e5d8b3ea63cde9cf"} Oct 30 01:28:46.139860 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:46.139812 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk" event={"ID":"6b19732a-da65-4b29-a2e7-a2017ce9b815","Type":"ContainerDied","Data":"716c65e99388430997cb21491a77918f2499a42734985c9a352d76b3442018cc"} Oct 30 01:28:46.139860 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:46.139843 5300 scope.go:122] "RemoveContainer" containerID="5c00e6157e657fe49107d18a97654b150248268ce3896e00e5d8b3ea63cde9cf" Oct 30 01:28:46.150516 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:46.150491 5300 scope.go:122] "RemoveContainer" containerID="5c00e6157e657fe49107d18a97654b150248268ce3896e00e5d8b3ea63cde9cf" Oct 30 01:28:46.150812 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:28:46.150789 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c00e6157e657fe49107d18a97654b150248268ce3896e00e5d8b3ea63cde9cf\": container with ID starting with 5c00e6157e657fe49107d18a97654b150248268ce3896e00e5d8b3ea63cde9cf not found: ID does not exist" containerID="5c00e6157e657fe49107d18a97654b150248268ce3896e00e5d8b3ea63cde9cf" Oct 30 01:28:46.150860 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:46.150823 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c00e6157e657fe49107d18a97654b150248268ce3896e00e5d8b3ea63cde9cf"} err="failed to get container status \"5c00e6157e657fe49107d18a97654b150248268ce3896e00e5d8b3ea63cde9cf\": rpc error: code = NotFound desc = could not find container \"5c00e6157e657fe49107d18a97654b150248268ce3896e00e5d8b3ea63cde9cf\": container with ID starting with 5c00e6157e657fe49107d18a97654b150248268ce3896e00e5d8b3ea63cde9cf not found: ID does not exist" Oct 30 01:28:46.197721 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:46.197669 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk"] Oct 30 01:28:46.215845 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:46.215811 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-18a189-57d89979-ddjvk"] Oct 30 01:28:47.110185 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:47.110151 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:47.822943 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:47.822910 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b19732a-da65-4b29-a2e7-a2017ce9b815" path="/var/lib/kubelet/pods/6b19732a-da65-4b29-a2e7-a2017ce9b815/volumes" Oct 30 01:28:48.587718 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:48.587687 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:49.795410 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:49.795377 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls"] Oct 30 01:28:49.795904 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:49.795594 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls" podUID="151eb97e-bb5b-44ca-89c9-07025f5cb2d6" containerName="maas-api" containerID="cri-o://3b21149e00670135dbded8b7806ddc1d41ec94555a0c0086284737b1be18cdd7" gracePeriod=30 Oct 30 01:28:49.846306 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:49.846277 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:50.095687 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.095661 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls" Oct 30 01:28:50.172834 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.172786 5300 generic.go:356] "Generic (PLEG): container finished" podID="151eb97e-bb5b-44ca-89c9-07025f5cb2d6" containerID="3b21149e00670135dbded8b7806ddc1d41ec94555a0c0086284737b1be18cdd7" exitCode=0 Oct 30 01:28:50.173028 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.172872 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls" event={"ID":"151eb97e-bb5b-44ca-89c9-07025f5cb2d6","Type":"ContainerDied","Data":"3b21149e00670135dbded8b7806ddc1d41ec94555a0c0086284737b1be18cdd7"} Oct 30 01:28:50.173028 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.172925 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls" event={"ID":"151eb97e-bb5b-44ca-89c9-07025f5cb2d6","Type":"ContainerDied","Data":"9a30bcc23d9ca15ef75b36a7d9e1e86452157626ddc96c13f4fb5486259abc04"} Oct 30 01:28:50.173028 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.172956 5300 scope.go:122] "RemoveContainer" containerID="3b21149e00670135dbded8b7806ddc1d41ec94555a0c0086284737b1be18cdd7" Oct 30 01:28:50.173028 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.172985 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls" Oct 30 01:28:50.187344 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.187305 5300 scope.go:122] "RemoveContainer" containerID="3b21149e00670135dbded8b7806ddc1d41ec94555a0c0086284737b1be18cdd7" Oct 30 01:28:50.187732 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:28:50.187702 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b21149e00670135dbded8b7806ddc1d41ec94555a0c0086284737b1be18cdd7\": container with ID starting with 3b21149e00670135dbded8b7806ddc1d41ec94555a0c0086284737b1be18cdd7 not found: ID does not exist" containerID="3b21149e00670135dbded8b7806ddc1d41ec94555a0c0086284737b1be18cdd7" Oct 30 01:28:50.187893 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.187747 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b21149e00670135dbded8b7806ddc1d41ec94555a0c0086284737b1be18cdd7"} err="failed to get container status \"3b21149e00670135dbded8b7806ddc1d41ec94555a0c0086284737b1be18cdd7\": rpc error: code = NotFound desc = could not find container \"3b21149e00670135dbded8b7806ddc1d41ec94555a0c0086284737b1be18cdd7\": container with ID starting with 3b21149e00670135dbded8b7806ddc1d41ec94555a0c0086284737b1be18cdd7 not found: ID does not exist" Oct 30 01:28:50.190802 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.190742 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-openshift-service-ca\") pod \"151eb97e-bb5b-44ca-89c9-07025f5cb2d6\" (UID: \"151eb97e-bb5b-44ca-89c9-07025f5cb2d6\") " Oct 30 01:28:50.191168 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.191138 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-openshift-service-ca" pod "151eb97e-bb5b-44ca-89c9-07025f5cb2d6" (UID: "151eb97e-bb5b-44ca-89c9-07025f5cb2d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:50.191464 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.191446 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-kube-api-access-4xvfv\" (UniqueName: \"kubernetes.io/projected/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-kube-api-access-4xvfv\") pod \"151eb97e-bb5b-44ca-89c9-07025f5cb2d6\" (UID: \"151eb97e-bb5b-44ca-89c9-07025f5cb2d6\") " Oct 30 01:28:50.191648 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.191624 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-metrics-certs\") pod \"151eb97e-bb5b-44ca-89c9-07025f5cb2d6\" (UID: \"151eb97e-bb5b-44ca-89c9-07025f5cb2d6\") " Oct 30 01:28:50.192090 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.192072 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-maas-api-tls\") pod \"151eb97e-bb5b-44ca-89c9-07025f5cb2d6\" (UID: \"151eb97e-bb5b-44ca-89c9-07025f5cb2d6\") " Oct 30 01:28:50.193226 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.193193 5300 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-openshift-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:50.232641 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.231992 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-metrics-certs" pod "151eb97e-bb5b-44ca-89c9-07025f5cb2d6" (UID: "151eb97e-bb5b-44ca-89c9-07025f5cb2d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:28:50.246473 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.246428 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-kube-api-access-4xvfv" pod "151eb97e-bb5b-44ca-89c9-07025f5cb2d6" (UID: "151eb97e-bb5b-44ca-89c9-07025f5cb2d6"). InnerVolumeSpecName "kube-api-access-4xvfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:50.297822 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.261961 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-maas-api-tls" pod "151eb97e-bb5b-44ca-89c9-07025f5cb2d6" (UID: "151eb97e-bb5b-44ca-89c9-07025f5cb2d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:28:50.301284 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.301017 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-maas-api-tls\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:50.301284 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.301051 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xvfv\" (UniqueName: \"kubernetes.io/projected/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-kube-api-access-4xvfv\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:50.301284 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.301082 5300 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/151eb97e-bb5b-44ca-89c9-07025f5cb2d6-metrics-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:50.411236 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.411151 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:50.495749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.495723 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:50.526690 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.526657 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls"] Oct 30 01:28:50.530153 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.530125 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ba3d7b-6d5f7597f9-sdlls"] Oct 30 01:28:50.621195 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.621161 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:50.743553 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.743518 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj"] Oct 30 01:28:50.743812 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.743774 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj" podUID="ffd43904-bf14-4a9c-a752-d18c1fcff864" containerName="maas-api" containerID="cri-o://80abc9bf42fd6b7b9d2af876a523c782b588b9a94e5af7de55cadb5d68a92713" gracePeriod=30 Oct 30 01:28:50.882816 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:50.880884 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:51.017656 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.015594 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:51.135111 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.135072 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:51.173458 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.173432 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj" Oct 30 01:28:51.182983 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.182943 5300 generic.go:356] "Generic (PLEG): container finished" podID="ffd43904-bf14-4a9c-a752-d18c1fcff864" containerID="80abc9bf42fd6b7b9d2af876a523c782b588b9a94e5af7de55cadb5d68a92713" exitCode=0 Oct 30 01:28:51.183159 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.182989 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj" event={"ID":"ffd43904-bf14-4a9c-a752-d18c1fcff864","Type":"ContainerDied","Data":"80abc9bf42fd6b7b9d2af876a523c782b588b9a94e5af7de55cadb5d68a92713"} Oct 30 01:28:51.183159 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.183057 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj" event={"ID":"ffd43904-bf14-4a9c-a752-d18c1fcff864","Type":"ContainerDied","Data":"d15f178ec6ed6de899349737f31c5a6f0cc7181462dc46e0e970ce19a0e09298"} Oct 30 01:28:51.183159 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.183000 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj" Oct 30 01:28:51.183159 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.183090 5300 scope.go:122] "RemoveContainer" containerID="80abc9bf42fd6b7b9d2af876a523c782b588b9a94e5af7de55cadb5d68a92713" Oct 30 01:28:51.195781 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.195745 5300 scope.go:122] "RemoveContainer" containerID="80abc9bf42fd6b7b9d2af876a523c782b588b9a94e5af7de55cadb5d68a92713" Oct 30 01:28:51.196178 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:28:51.196145 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80abc9bf42fd6b7b9d2af876a523c782b588b9a94e5af7de55cadb5d68a92713\": container with ID starting with 80abc9bf42fd6b7b9d2af876a523c782b588b9a94e5af7de55cadb5d68a92713 not found: ID does not exist" containerID="80abc9bf42fd6b7b9d2af876a523c782b588b9a94e5af7de55cadb5d68a92713" Oct 30 01:28:51.196369 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.196192 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80abc9bf42fd6b7b9d2af876a523c782b588b9a94e5af7de55cadb5d68a92713"} err="failed to get container status \"80abc9bf42fd6b7b9d2af876a523c782b588b9a94e5af7de55cadb5d68a92713\": rpc error: code = NotFound desc = could not find container \"80abc9bf42fd6b7b9d2af876a523c782b588b9a94e5af7de55cadb5d68a92713\": container with ID starting with 80abc9bf42fd6b7b9d2af876a523c782b588b9a94e5af7de55cadb5d68a92713 not found: ID does not exist" Oct 30 01:28:51.237302 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.237253 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:51.355470 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.355375 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffd43904-bf14-4a9c-a752-d18c1fcff864-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd43904-bf14-4a9c-a752-d18c1fcff864-openshift-service-ca\") pod \"ffd43904-bf14-4a9c-a752-d18c1fcff864\" (UID: \"ffd43904-bf14-4a9c-a752-d18c1fcff864\") " Oct 30 01:28:51.355664 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.355484 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffd43904-bf14-4a9c-a752-d18c1fcff864-kube-api-access-bxvwd\" (UniqueName: \"kubernetes.io/projected/ffd43904-bf14-4a9c-a752-d18c1fcff864-kube-api-access-bxvwd\") pod \"ffd43904-bf14-4a9c-a752-d18c1fcff864\" (UID: \"ffd43904-bf14-4a9c-a752-d18c1fcff864\") " Oct 30 01:28:51.355664 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.355544 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffd43904-bf14-4a9c-a752-d18c1fcff864-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffd43904-bf14-4a9c-a752-d18c1fcff864-metrics-certs\") pod \"ffd43904-bf14-4a9c-a752-d18c1fcff864\" (UID: \"ffd43904-bf14-4a9c-a752-d18c1fcff864\") " Oct 30 01:28:51.355819 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.355664 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffd43904-bf14-4a9c-a752-d18c1fcff864-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffd43904-bf14-4a9c-a752-d18c1fcff864-maas-api-tls\") pod \"ffd43904-bf14-4a9c-a752-d18c1fcff864\" (UID: \"ffd43904-bf14-4a9c-a752-d18c1fcff864\") " Oct 30 01:28:51.356042 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.356010 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd43904-bf14-4a9c-a752-d18c1fcff864-openshift-service-ca" pod "ffd43904-bf14-4a9c-a752-d18c1fcff864" (UID: "ffd43904-bf14-4a9c-a752-d18c1fcff864"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:51.356534 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.356513 5300 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd43904-bf14-4a9c-a752-d18c1fcff864-openshift-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:51.358471 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.358441 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffd43904-bf14-4a9c-a752-d18c1fcff864-maas-api-tls" pod "ffd43904-bf14-4a9c-a752-d18c1fcff864" (UID: "ffd43904-bf14-4a9c-a752-d18c1fcff864"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:28:51.359696 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.359671 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffd43904-bf14-4a9c-a752-d18c1fcff864-metrics-certs" pod "ffd43904-bf14-4a9c-a752-d18c1fcff864" (UID: "ffd43904-bf14-4a9c-a752-d18c1fcff864"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:28:51.391060 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.390976 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd43904-bf14-4a9c-a752-d18c1fcff864-kube-api-access-bxvwd" pod "ffd43904-bf14-4a9c-a752-d18c1fcff864" (UID: "ffd43904-bf14-4a9c-a752-d18c1fcff864"). InnerVolumeSpecName "kube-api-access-bxvwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:51.422877 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.422829 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:51.460211 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.460174 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxvwd\" (UniqueName: \"kubernetes.io/projected/ffd43904-bf14-4a9c-a752-d18c1fcff864-kube-api-access-bxvwd\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:51.461153 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.461126 5300 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffd43904-bf14-4a9c-a752-d18c1fcff864-metrics-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:51.461153 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.461155 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffd43904-bf14-4a9c-a752-d18c1fcff864-maas-api-tls\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:51.528964 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.528931 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:51.543252 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.543210 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj"] Oct 30 01:28:51.549426 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.549395 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-e2d6d3-d565dfb69-8j4jj"] Oct 30 01:28:51.575954 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:28:51.575909 5300 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd43904_bf14_4a9c_a752_d18c1fcff864.slice/crio-d15f178ec6ed6de899349737f31c5a6f0cc7181462dc46e0e970ce19a0e09298\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd43904_bf14_4a9c_a752_d18c1fcff864.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:28:51.604784 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:28:51.603370 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:28:51.637409 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.637321 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:51.737969 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.737936 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:51.825384 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.825349 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="151eb97e-bb5b-44ca-89c9-07025f5cb2d6" path="/var/lib/kubelet/pods/151eb97e-bb5b-44ca-89c9-07025f5cb2d6/volumes" Oct 30 01:28:51.825945 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:51.825928 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffd43904-bf14-4a9c-a752-d18c1fcff864" path="/var/lib/kubelet/pods/ffd43904-bf14-4a9c-a752-d18c1fcff864/volumes" Oct 30 01:28:52.079416 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:52.079384 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:52.960193 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:52.960158 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:53.102321 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:53.102294 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:54.128107 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:54.128072 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:55.413299 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:55.413266 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c"] Oct 30 01:28:55.413774 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:55.413537 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c" podUID="5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001" containerName="maas-api" containerID="cri-o://134ef7a3e1a73f5fbdeadf539f5bf587d59da39f77557d6adfe7ff785db1ebb0" gracePeriod=30 Oct 30 01:28:55.471060 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:55.471019 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:55.481992 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:55.481964 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc"] Oct 30 01:28:55.729454 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:55.729429 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:55.752062 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:55.752026 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c" Oct 30 01:28:55.841886 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:55.841843 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:55.844170 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:55.844134 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-openshift-service-ca\") pod \"5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001\" (UID: \"5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001\") " Oct 30 01:28:55.844324 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:55.844195 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-kube-api-access-mncbk\" (UniqueName: \"kubernetes.io/projected/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-kube-api-access-mncbk\") pod \"5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001\" (UID: \"5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001\") " Oct 30 01:28:55.844324 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:55.844275 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-maas-api-tls\") pod \"5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001\" (UID: \"5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001\") " Oct 30 01:28:55.844481 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:55.844462 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-metrics-certs\") pod \"5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001\" (UID: \"5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001\") " Oct 30 01:28:55.844579 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:55.844551 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-openshift-service-ca" pod "5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001" (UID: "5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:55.845276 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:55.845248 5300 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-openshift-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:55.856938 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:55.856904 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-maas-api-tls" pod "5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001" (UID: "5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:28:55.857069 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:55.856935 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-metrics-certs" pod "5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001" (UID: "5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:28:55.857069 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:55.856944 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-kube-api-access-mncbk" pod "5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001" (UID: "5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001"). InnerVolumeSpecName "kube-api-access-mncbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:55.947287 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:55.947247 5300 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-metrics-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:55.947287 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:55.947280 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mncbk\" (UniqueName: \"kubernetes.io/projected/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-kube-api-access-mncbk\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:55.947530 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:55.947295 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001-maas-api-tls\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:28:56.229698 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:56.229659 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:56.250968 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:56.250935 5300 generic.go:356] "Generic (PLEG): container finished" podID="5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001" containerID="134ef7a3e1a73f5fbdeadf539f5bf587d59da39f77557d6adfe7ff785db1ebb0" exitCode=0 Oct 30 01:28:56.251168 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:56.251087 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c" event={"ID":"5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001","Type":"ContainerDied","Data":"134ef7a3e1a73f5fbdeadf539f5bf587d59da39f77557d6adfe7ff785db1ebb0"} Oct 30 01:28:56.251168 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:56.251125 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c" event={"ID":"5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001","Type":"ContainerDied","Data":"290fae8e46f84e151a239b25214c2ba58d9b0621bdd3d45b081211d059ae7f34"} Oct 30 01:28:56.251168 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:56.251145 5300 scope.go:122] "RemoveContainer" containerID="134ef7a3e1a73f5fbdeadf539f5bf587d59da39f77557d6adfe7ff785db1ebb0" Oct 30 01:28:56.251349 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:56.251259 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c" Oct 30 01:28:56.275329 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:56.273897 5300 scope.go:122] "RemoveContainer" containerID="134ef7a3e1a73f5fbdeadf539f5bf587d59da39f77557d6adfe7ff785db1ebb0" Oct 30 01:28:56.275969 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:28:56.275905 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134ef7a3e1a73f5fbdeadf539f5bf587d59da39f77557d6adfe7ff785db1ebb0\": container with ID starting with 134ef7a3e1a73f5fbdeadf539f5bf587d59da39f77557d6adfe7ff785db1ebb0 not found: ID does not exist" containerID="134ef7a3e1a73f5fbdeadf539f5bf587d59da39f77557d6adfe7ff785db1ebb0" Oct 30 01:28:56.275969 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:56.275945 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134ef7a3e1a73f5fbdeadf539f5bf587d59da39f77557d6adfe7ff785db1ebb0"} err="failed to get container status \"134ef7a3e1a73f5fbdeadf539f5bf587d59da39f77557d6adfe7ff785db1ebb0\": rpc error: code = NotFound desc = could not find container \"134ef7a3e1a73f5fbdeadf539f5bf587d59da39f77557d6adfe7ff785db1ebb0\": container with ID starting with 134ef7a3e1a73f5fbdeadf539f5bf587d59da39f77557d6adfe7ff785db1ebb0 not found: ID does not exist" Oct 30 01:28:56.317219 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:56.317187 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c"] Oct 30 01:28:56.328223 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:56.328190 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7f0263-58994d7f8b-xw99c"] Oct 30 01:28:56.362751 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:56.362711 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:56.774873 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:56.774839 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:57.824900 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:57.824865 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001" path="/var/lib/kubelet/pods/5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001/volumes" Oct 30 01:28:58.087249 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:58.087158 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:58.452152 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:58.452063 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:58.545140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:58.545102 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:58.772037 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:58.772003 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k"] Oct 30 01:28:58.772492 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:58.772463 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" podUID="ce797d22-c33f-4853-8c58-e014cd26bf5b" containerName="istio-proxy" containerID="cri-o://b1ca93273d8c925fca792624f4a23ffa5e1f986dc617b88f4446775f366a1446" gracePeriod=30 Oct 30 01:28:58.796189 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:58.796162 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:58.981715 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:58.981663 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:59.072630 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:59.072540 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:59.253917 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:59.253871 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp"] Oct 30 01:28:59.254186 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:59.254157 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" podUID="d13883c2-ec67-4526-985d-f22c6673853f" containerName="istio-proxy" containerID="cri-o://970d0565450e21b0ef7693b3afc397a52d15f09b9a33c0954732b2b1f66d1d6a" gracePeriod=30 Oct 30 01:28:59.280303 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:59.280271 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:59.396827 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:59.396693 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp"] Oct 30 01:28:59.397008 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:59.396946 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" podUID="1455aae2-7d1f-4788-9a94-7469bee17334" containerName="istio-proxy" containerID="cri-o://b6db75e823b319ca70be02011a1ea7b9a20ba9e92aeb44a5f3c935ad42adc3ee" gracePeriod=30 Oct 30 01:28:59.408291 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:59.408259 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:59.968819 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:28:59.968754 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:00.482591 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:00.482551 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc" podUID="26bb7124-a395-4690-a7dc-9d2236e9a083" containerName="payload-processing" containerID="cri-o://f2a7b43c216409deae6851451548d7cfbca94c71979098835da05f49a4d859d1" gracePeriod=25 Oct 30 01:29:00.787327 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:00.787300 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc" Oct 30 01:29:00.880518 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:00.880482 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26bb7124-a395-4690-a7dc-9d2236e9a083-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26bb7124-a395-4690-a7dc-9d2236e9a083-plugins-config-volume\") pod \"26bb7124-a395-4690-a7dc-9d2236e9a083\" (UID: \"26bb7124-a395-4690-a7dc-9d2236e9a083\") " Oct 30 01:29:00.880720 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:00.880651 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26bb7124-a395-4690-a7dc-9d2236e9a083-kube-api-access-6lfh9\" (UniqueName: \"kubernetes.io/projected/26bb7124-a395-4690-a7dc-9d2236e9a083-kube-api-access-6lfh9\") pod \"26bb7124-a395-4690-a7dc-9d2236e9a083\" (UID: \"26bb7124-a395-4690-a7dc-9d2236e9a083\") " Oct 30 01:29:00.881014 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:00.880981 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26bb7124-a395-4690-a7dc-9d2236e9a083-plugins-config-volume" pod "26bb7124-a395-4690-a7dc-9d2236e9a083" (UID: "26bb7124-a395-4690-a7dc-9d2236e9a083"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:00.882113 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:00.882089 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26bb7124-a395-4690-a7dc-9d2236e9a083-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:00.884332 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:00.884305 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26bb7124-a395-4690-a7dc-9d2236e9a083-kube-api-access-6lfh9" pod "26bb7124-a395-4690-a7dc-9d2236e9a083" (UID: "26bb7124-a395-4690-a7dc-9d2236e9a083"). InnerVolumeSpecName "kube-api-access-6lfh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:00.983564 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:00.983530 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lfh9\" (UniqueName: \"kubernetes.io/projected/26bb7124-a395-4690-a7dc-9d2236e9a083-kube-api-access-6lfh9\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:01.336741 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:01.336709 5300 generic.go:356] "Generic (PLEG): container finished" podID="26bb7124-a395-4690-a7dc-9d2236e9a083" containerID="f2a7b43c216409deae6851451548d7cfbca94c71979098835da05f49a4d859d1" exitCode=0 Oct 30 01:29:01.336941 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:01.336789 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc" Oct 30 01:29:01.336941 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:01.336789 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc" event={"ID":"26bb7124-a395-4690-a7dc-9d2236e9a083","Type":"ContainerDied","Data":"f2a7b43c216409deae6851451548d7cfbca94c71979098835da05f49a4d859d1"} Oct 30 01:29:01.336941 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:01.336917 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc" event={"ID":"26bb7124-a395-4690-a7dc-9d2236e9a083","Type":"ContainerDied","Data":"9cc0d93aafaf3305ed85d02fc531292300b870d51b4909e3e915b523ad868a36"} Oct 30 01:29:01.336941 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:01.336940 5300 scope.go:122] "RemoveContainer" containerID="f2a7b43c216409deae6851451548d7cfbca94c71979098835da05f49a4d859d1" Oct 30 01:29:01.349749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:01.349730 5300 scope.go:122] "RemoveContainer" containerID="f2a7b43c216409deae6851451548d7cfbca94c71979098835da05f49a4d859d1" Oct 30 01:29:01.350133 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:29:01.350105 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2a7b43c216409deae6851451548d7cfbca94c71979098835da05f49a4d859d1\": container with ID starting with f2a7b43c216409deae6851451548d7cfbca94c71979098835da05f49a4d859d1 not found: ID does not exist" containerID="f2a7b43c216409deae6851451548d7cfbca94c71979098835da05f49a4d859d1" Oct 30 01:29:01.350220 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:01.350143 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2a7b43c216409deae6851451548d7cfbca94c71979098835da05f49a4d859d1"} err="failed to get container status \"f2a7b43c216409deae6851451548d7cfbca94c71979098835da05f49a4d859d1\": rpc error: code = NotFound desc = could not find container \"f2a7b43c216409deae6851451548d7cfbca94c71979098835da05f49a4d859d1\": container with ID starting with f2a7b43c216409deae6851451548d7cfbca94c71979098835da05f49a4d859d1 not found: ID does not exist" Oct 30 01:29:01.394223 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:01.394196 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc"] Oct 30 01:29:01.408382 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:01.408358 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eaf753-75dd8cdcd8-rrlcc"] Oct 30 01:29:01.640292 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:01.640171 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:01.827185 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:01.827153 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26bb7124-a395-4690-a7dc-9d2236e9a083" path="/var/lib/kubelet/pods/26bb7124-a395-4690-a7dc-9d2236e9a083/volumes" Oct 30 01:29:01.838212 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:01.838183 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:01.935578 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:01.935484 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:03.180974 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:03.180941 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:03.277101 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:03.277071 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:03.379848 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:03.379806 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:03.646386 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:03.646353 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:03.693637 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:03.693605 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:29:03.693863 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:03.693660 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" podUID="d13883c2-ec67-4526-985d-f22c6673853f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:29:03.743005 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:03.742949 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:04.115672 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.115648 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 01:29:04.238729 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.238688 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-workload-socket\") pod \"ce797d22-c33f-4853-8c58-e014cd26bf5b\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " Oct 30 01:29:04.239749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.238877 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce797d22-c33f-4853-8c58-e014cd26bf5b-kube-api-access-4h4gc\" (UniqueName: \"kubernetes.io/projected/ce797d22-c33f-4853-8c58-e014cd26bf5b-kube-api-access-4h4gc\") pod \"ce797d22-c33f-4853-8c58-e014cd26bf5b\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " Oct 30 01:29:04.239749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.239000 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-data\") pod \"ce797d22-c33f-4853-8c58-e014cd26bf5b\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " Oct 30 01:29:04.239749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.239041 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-envoy\") pod \"ce797d22-c33f-4853-8c58-e014cd26bf5b\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " Oct 30 01:29:04.239749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.239572 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-workload-certs\") pod \"ce797d22-c33f-4853-8c58-e014cd26bf5b\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " Oct 30 01:29:04.239749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.239650 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-credential-socket\") pod \"ce797d22-c33f-4853-8c58-e014cd26bf5b\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " Oct 30 01:29:04.239749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.239721 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-podinfo\") pod \"ce797d22-c33f-4853-8c58-e014cd26bf5b\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " Oct 30 01:29:04.240508 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.239805 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce797d22-c33f-4853-8c58-e014cd26bf5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce797d22-c33f-4853-8c58-e014cd26bf5b-istiod-ca-cert\") pod \"ce797d22-c33f-4853-8c58-e014cd26bf5b\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " Oct 30 01:29:04.240508 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.239841 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-token\" (UniqueName: \"kubernetes.io/projected/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-token\") pod \"ce797d22-c33f-4853-8c58-e014cd26bf5b\" (UID: \"ce797d22-c33f-4853-8c58-e014cd26bf5b\") " Oct 30 01:29:04.242301 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.241011 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-workload-socket" pod "ce797d22-c33f-4853-8c58-e014cd26bf5b" (UID: "ce797d22-c33f-4853-8c58-e014cd26bf5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:04.242301 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.241190 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-workload-certs" pod "ce797d22-c33f-4853-8c58-e014cd26bf5b" (UID: "ce797d22-c33f-4853-8c58-e014cd26bf5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:04.242301 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.241586 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-data" pod "ce797d22-c33f-4853-8c58-e014cd26bf5b" (UID: "ce797d22-c33f-4853-8c58-e014cd26bf5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:04.242301 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.242128 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce797d22-c33f-4853-8c58-e014cd26bf5b-istiod-ca-cert" pod "ce797d22-c33f-4853-8c58-e014cd26bf5b" (UID: "ce797d22-c33f-4853-8c58-e014cd26bf5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:04.242301 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.242264 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-credential-socket" pod "ce797d22-c33f-4853-8c58-e014cd26bf5b" (UID: "ce797d22-c33f-4853-8c58-e014cd26bf5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:04.245890 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.245841 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-podinfo" pod "ce797d22-c33f-4853-8c58-e014cd26bf5b" (UID: "ce797d22-c33f-4853-8c58-e014cd26bf5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:29:04.246017 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.245995 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-token" pod "ce797d22-c33f-4853-8c58-e014cd26bf5b" (UID: "ce797d22-c33f-4853-8c58-e014cd26bf5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:04.246085 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.246042 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-envoy" pod "ce797d22-c33f-4853-8c58-e014cd26bf5b" (UID: "ce797d22-c33f-4853-8c58-e014cd26bf5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:04.246881 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.246838 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce797d22-c33f-4853-8c58-e014cd26bf5b-kube-api-access-4h4gc" pod "ce797d22-c33f-4853-8c58-e014cd26bf5b" (UID: "ce797d22-c33f-4853-8c58-e014cd26bf5b"). InnerVolumeSpecName "kube-api-access-4h4gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:04.342160 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.342129 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.342160 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.342156 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.342160 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.342167 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.342366 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.342175 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce797d22-c33f-4853-8c58-e014cd26bf5b-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.342366 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.342184 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.342366 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.342191 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.342366 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.342199 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h4gc\" (UniqueName: \"kubernetes.io/projected/ce797d22-c33f-4853-8c58-e014cd26bf5b-kube-api-access-4h4gc\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.342366 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.342208 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.342366 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.342216 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce797d22-c33f-4853-8c58-e014cd26bf5b-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.365234 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.365203 5300 generic.go:356] "Generic (PLEG): container finished" podID="ce797d22-c33f-4853-8c58-e014cd26bf5b" containerID="b1ca93273d8c925fca792624f4a23ffa5e1f986dc617b88f4446775f366a1446" exitCode=0 Oct 30 01:29:04.365396 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.365286 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" Oct 30 01:29:04.365396 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.365289 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" event={"ID":"ce797d22-c33f-4853-8c58-e014cd26bf5b","Type":"ContainerDied","Data":"b1ca93273d8c925fca792624f4a23ffa5e1f986dc617b88f4446775f366a1446"} Oct 30 01:29:04.365518 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.365433 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k" event={"ID":"ce797d22-c33f-4853-8c58-e014cd26bf5b","Type":"ContainerDied","Data":"9d3ae85541b8b065d6acb019f3ea664ee473292e7c44767f9e5b096802f4ccab"} Oct 30 01:29:04.365518 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.365456 5300 scope.go:122] "RemoveContainer" containerID="b1ca93273d8c925fca792624f4a23ffa5e1f986dc617b88f4446775f366a1446" Oct 30 01:29:04.378344 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.378321 5300 scope.go:122] "RemoveContainer" containerID="b6c96873414c39606d49caf52f7f34febe7ae307d4bc695f36d02bbf62dae394" Oct 30 01:29:04.397516 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.397496 5300 scope.go:122] "RemoveContainer" containerID="b1ca93273d8c925fca792624f4a23ffa5e1f986dc617b88f4446775f366a1446" Oct 30 01:29:04.397918 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:29:04.397895 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1ca93273d8c925fca792624f4a23ffa5e1f986dc617b88f4446775f366a1446\": container with ID starting with b1ca93273d8c925fca792624f4a23ffa5e1f986dc617b88f4446775f366a1446 not found: ID does not exist" containerID="b1ca93273d8c925fca792624f4a23ffa5e1f986dc617b88f4446775f366a1446" Oct 30 01:29:04.398020 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.397931 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ca93273d8c925fca792624f4a23ffa5e1f986dc617b88f4446775f366a1446"} err="failed to get container status \"b1ca93273d8c925fca792624f4a23ffa5e1f986dc617b88f4446775f366a1446\": rpc error: code = NotFound desc = could not find container \"b1ca93273d8c925fca792624f4a23ffa5e1f986dc617b88f4446775f366a1446\": container with ID starting with b1ca93273d8c925fca792624f4a23ffa5e1f986dc617b88f4446775f366a1446 not found: ID does not exist" Oct 30 01:29:04.398020 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.397957 5300 scope.go:122] "RemoveContainer" containerID="b6c96873414c39606d49caf52f7f34febe7ae307d4bc695f36d02bbf62dae394" Oct 30 01:29:04.398299 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:29:04.398275 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6c96873414c39606d49caf52f7f34febe7ae307d4bc695f36d02bbf62dae394\": container with ID starting with b6c96873414c39606d49caf52f7f34febe7ae307d4bc695f36d02bbf62dae394 not found: ID does not exist" containerID="b6c96873414c39606d49caf52f7f34febe7ae307d4bc695f36d02bbf62dae394" Oct 30 01:29:04.398357 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.398314 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6c96873414c39606d49caf52f7f34febe7ae307d4bc695f36d02bbf62dae394"} err="failed to get container status \"b6c96873414c39606d49caf52f7f34febe7ae307d4bc695f36d02bbf62dae394\": rpc error: code = NotFound desc = could not find container \"b6c96873414c39606d49caf52f7f34febe7ae307d4bc695f36d02bbf62dae394\": container with ID starting with b6c96873414c39606d49caf52f7f34febe7ae307d4bc695f36d02bbf62dae394 not found: ID does not exist" Oct 30 01:29:04.427387 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.427353 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k"] Oct 30 01:29:04.430853 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.430827 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ba3d7b-openshift-default-5d87b99755-vrk6k"] Oct 30 01:29:04.531087 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.531069 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:29:04.647563 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.646836 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d13883c2-ec67-4526-985d-f22c6673853f-kube-api-access-5shdr\" (UniqueName: \"kubernetes.io/projected/d13883c2-ec67-4526-985d-f22c6673853f-kube-api-access-5shdr\") pod \"d13883c2-ec67-4526-985d-f22c6673853f\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " Oct 30 01:29:04.647563 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.646925 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-workload-socket\") pod \"d13883c2-ec67-4526-985d-f22c6673853f\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " Oct 30 01:29:04.647563 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.646992 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d13883c2-ec67-4526-985d-f22c6673853f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d13883c2-ec67-4526-985d-f22c6673853f-istiod-ca-cert\") pod \"d13883c2-ec67-4526-985d-f22c6673853f\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " Oct 30 01:29:04.647563 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.647105 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-workload-certs\") pod \"d13883c2-ec67-4526-985d-f22c6673853f\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " Oct 30 01:29:04.647563 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.647182 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-credential-socket\") pod \"d13883c2-ec67-4526-985d-f22c6673853f\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " Oct 30 01:29:04.647563 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.647222 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d13883c2-ec67-4526-985d-f22c6673853f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d13883c2-ec67-4526-985d-f22c6673853f-istio-podinfo\") pod \"d13883c2-ec67-4526-985d-f22c6673853f\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " Oct 30 01:29:04.647563 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.647300 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-istio-envoy\") pod \"d13883c2-ec67-4526-985d-f22c6673853f\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " Oct 30 01:29:04.647563 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.647336 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-istio-data\") pod \"d13883c2-ec67-4526-985d-f22c6673853f\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " Oct 30 01:29:04.649513 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.649477 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-workload-certs" pod "d13883c2-ec67-4526-985d-f22c6673853f" (UID: "d13883c2-ec67-4526-985d-f22c6673853f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:04.650374 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.650345 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-credential-socket" pod "d13883c2-ec67-4526-985d-f22c6673853f" (UID: "d13883c2-ec67-4526-985d-f22c6673853f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:04.650718 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.650693 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-istio-data" pod "d13883c2-ec67-4526-985d-f22c6673853f" (UID: "d13883c2-ec67-4526-985d-f22c6673853f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:04.651298 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.650910 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d13883c2-ec67-4526-985d-f22c6673853f-istiod-ca-cert" pod "d13883c2-ec67-4526-985d-f22c6673853f" (UID: "d13883c2-ec67-4526-985d-f22c6673853f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:04.651298 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.651144 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d13883c2-ec67-4526-985d-f22c6673853f-istio-token\" (UniqueName: \"kubernetes.io/projected/d13883c2-ec67-4526-985d-f22c6673853f-istio-token\") pod \"d13883c2-ec67-4526-985d-f22c6673853f\" (UID: \"d13883c2-ec67-4526-985d-f22c6673853f\") " Oct 30 01:29:04.651875 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.651823 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-workload-socket" pod "d13883c2-ec67-4526-985d-f22c6673853f" (UID: "d13883c2-ec67-4526-985d-f22c6673853f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:04.651961 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.651916 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.651961 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.651940 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.651961 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.651954 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.652139 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.651968 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d13883c2-ec67-4526-985d-f22c6673853f-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.652557 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.652532 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13883c2-ec67-4526-985d-f22c6673853f-kube-api-access-5shdr" pod "d13883c2-ec67-4526-985d-f22c6673853f" (UID: "d13883c2-ec67-4526-985d-f22c6673853f"). InnerVolumeSpecName "kube-api-access-5shdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:04.655915 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.655873 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-istio-envoy" pod "d13883c2-ec67-4526-985d-f22c6673853f" (UID: "d13883c2-ec67-4526-985d-f22c6673853f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:04.657143 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.657123 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d13883c2-ec67-4526-985d-f22c6673853f-istio-podinfo" pod "d13883c2-ec67-4526-985d-f22c6673853f" (UID: "d13883c2-ec67-4526-985d-f22c6673853f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:29:04.657462 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.657432 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13883c2-ec67-4526-985d-f22c6673853f-istio-token" pod "d13883c2-ec67-4526-985d-f22c6673853f" (UID: "d13883c2-ec67-4526-985d-f22c6673853f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:04.707221 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.707192 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:29:04.754303 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.754273 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d13883c2-ec67-4526-985d-f22c6673853f-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.754521 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.754496 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.754521 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.754522 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d13883c2-ec67-4526-985d-f22c6673853f-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.754712 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.754536 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5shdr\" (UniqueName: \"kubernetes.io/projected/d13883c2-ec67-4526-985d-f22c6673853f-kube-api-access-5shdr\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.754712 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.754550 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13883c2-ec67-4526-985d-f22c6673853f-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.855500 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.855143 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1455aae2-7d1f-4788-9a94-7469bee17334-kube-api-access-qxzrs\" (UniqueName: \"kubernetes.io/projected/1455aae2-7d1f-4788-9a94-7469bee17334-kube-api-access-qxzrs\") pod \"1455aae2-7d1f-4788-9a94-7469bee17334\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " Oct 30 01:29:04.855500 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.855192 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-istio-data\") pod \"1455aae2-7d1f-4788-9a94-7469bee17334\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " Oct 30 01:29:04.855500 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.855329 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1455aae2-7d1f-4788-9a94-7469bee17334-istio-token\" (UniqueName: \"kubernetes.io/projected/1455aae2-7d1f-4788-9a94-7469bee17334-istio-token\") pod \"1455aae2-7d1f-4788-9a94-7469bee17334\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " Oct 30 01:29:04.855500 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.855394 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-workload-socket\") pod \"1455aae2-7d1f-4788-9a94-7469bee17334\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " Oct 30 01:29:04.855500 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.855446 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-workload-certs\") pod \"1455aae2-7d1f-4788-9a94-7469bee17334\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " Oct 30 01:29:04.855500 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.855475 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1455aae2-7d1f-4788-9a94-7469bee17334-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1455aae2-7d1f-4788-9a94-7469bee17334-istio-podinfo\") pod \"1455aae2-7d1f-4788-9a94-7469bee17334\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " Oct 30 01:29:04.855985 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.855531 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-istio-data" pod "1455aae2-7d1f-4788-9a94-7469bee17334" (UID: "1455aae2-7d1f-4788-9a94-7469bee17334"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:04.855985 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.855602 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-istio-envoy\") pod \"1455aae2-7d1f-4788-9a94-7469bee17334\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " Oct 30 01:29:04.855985 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.855639 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1455aae2-7d1f-4788-9a94-7469bee17334-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1455aae2-7d1f-4788-9a94-7469bee17334-istiod-ca-cert\") pod \"1455aae2-7d1f-4788-9a94-7469bee17334\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " Oct 30 01:29:04.855985 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.855724 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-credential-socket\") pod \"1455aae2-7d1f-4788-9a94-7469bee17334\" (UID: \"1455aae2-7d1f-4788-9a94-7469bee17334\") " Oct 30 01:29:04.855985 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.855753 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-workload-socket" pod "1455aae2-7d1f-4788-9a94-7469bee17334" (UID: "1455aae2-7d1f-4788-9a94-7469bee17334"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:04.856337 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.856295 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1455aae2-7d1f-4788-9a94-7469bee17334-istiod-ca-cert" pod "1455aae2-7d1f-4788-9a94-7469bee17334" (UID: "1455aae2-7d1f-4788-9a94-7469bee17334"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:04.856978 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.856468 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-credential-socket" pod "1455aae2-7d1f-4788-9a94-7469bee17334" (UID: "1455aae2-7d1f-4788-9a94-7469bee17334"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:04.856978 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.856675 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1455aae2-7d1f-4788-9a94-7469bee17334-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.856978 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.856696 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.856978 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.856712 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.856978 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.856724 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.856978 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.856946 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-workload-certs" pod "1455aae2-7d1f-4788-9a94-7469bee17334" (UID: "1455aae2-7d1f-4788-9a94-7469bee17334"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:04.858226 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.858197 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1455aae2-7d1f-4788-9a94-7469bee17334-kube-api-access-qxzrs" pod "1455aae2-7d1f-4788-9a94-7469bee17334" (UID: "1455aae2-7d1f-4788-9a94-7469bee17334"). InnerVolumeSpecName "kube-api-access-qxzrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:04.858373 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.858340 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1455aae2-7d1f-4788-9a94-7469bee17334-istio-podinfo" pod "1455aae2-7d1f-4788-9a94-7469bee17334" (UID: "1455aae2-7d1f-4788-9a94-7469bee17334"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:29:04.859435 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.859409 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1455aae2-7d1f-4788-9a94-7469bee17334-istio-token" pod "1455aae2-7d1f-4788-9a94-7469bee17334" (UID: "1455aae2-7d1f-4788-9a94-7469bee17334"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:04.859859 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.859832 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-istio-envoy" pod "1455aae2-7d1f-4788-9a94-7469bee17334" (UID: "1455aae2-7d1f-4788-9a94-7469bee17334"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:04.958674 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.958638 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.958674 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.958671 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1455aae2-7d1f-4788-9a94-7469bee17334-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.958674 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.958681 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1455aae2-7d1f-4788-9a94-7469bee17334-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.958992 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.958693 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxzrs\" (UniqueName: \"kubernetes.io/projected/1455aae2-7d1f-4788-9a94-7469bee17334-kube-api-access-qxzrs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:04.958992 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:04.958702 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1455aae2-7d1f-4788-9a94-7469bee17334-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:05.334053 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:05.334023 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:05.375272 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:05.375240 5300 generic.go:356] "Generic (PLEG): container finished" podID="1455aae2-7d1f-4788-9a94-7469bee17334" containerID="b6db75e823b319ca70be02011a1ea7b9a20ba9e92aeb44a5f3c935ad42adc3ee" exitCode=0 Oct 30 01:29:05.375479 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:05.375298 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" Oct 30 01:29:05.375479 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:05.375322 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" event={"ID":"1455aae2-7d1f-4788-9a94-7469bee17334","Type":"ContainerDied","Data":"b6db75e823b319ca70be02011a1ea7b9a20ba9e92aeb44a5f3c935ad42adc3ee"} Oct 30 01:29:05.375479 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:05.375383 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp" event={"ID":"1455aae2-7d1f-4788-9a94-7469bee17334","Type":"ContainerDied","Data":"c06b8bc5785a4021677dbc806c370b192cfb2b07c5ca59474b6371261072ea71"} Oct 30 01:29:05.375479 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:05.375413 5300 scope.go:122] "RemoveContainer" containerID="b6db75e823b319ca70be02011a1ea7b9a20ba9e92aeb44a5f3c935ad42adc3ee" Oct 30 01:29:05.394790 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:05.391597 5300 generic.go:356] "Generic (PLEG): container finished" podID="d13883c2-ec67-4526-985d-f22c6673853f" containerID="970d0565450e21b0ef7693b3afc397a52d15f09b9a33c0954732b2b1f66d1d6a" exitCode=0 Oct 30 01:29:05.394790 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:05.391700 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" event={"ID":"d13883c2-ec67-4526-985d-f22c6673853f","Type":"ContainerDied","Data":"970d0565450e21b0ef7693b3afc397a52d15f09b9a33c0954732b2b1f66d1d6a"} Oct 30 01:29:05.394790 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:05.391752 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" event={"ID":"d13883c2-ec67-4526-985d-f22c6673853f","Type":"ContainerDied","Data":"87af2982273ebb5abeffa76234d0645a2ef8142d19d39a28db5ca2f8430f0dfe"} Oct 30 01:29:05.394790 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:05.391863 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp" Oct 30 01:29:05.418929 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:05.418897 5300 scope.go:122] "RemoveContainer" containerID="b6db75e823b319ca70be02011a1ea7b9a20ba9e92aeb44a5f3c935ad42adc3ee" Oct 30 01:29:05.419414 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:29:05.419388 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6db75e823b319ca70be02011a1ea7b9a20ba9e92aeb44a5f3c935ad42adc3ee\": container with ID starting with b6db75e823b319ca70be02011a1ea7b9a20ba9e92aeb44a5f3c935ad42adc3ee not found: ID does not exist" containerID="b6db75e823b319ca70be02011a1ea7b9a20ba9e92aeb44a5f3c935ad42adc3ee" Oct 30 01:29:05.419535 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:05.419423 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6db75e823b319ca70be02011a1ea7b9a20ba9e92aeb44a5f3c935ad42adc3ee"} err="failed to get container status \"b6db75e823b319ca70be02011a1ea7b9a20ba9e92aeb44a5f3c935ad42adc3ee\": rpc error: code = NotFound desc = could not find container \"b6db75e823b319ca70be02011a1ea7b9a20ba9e92aeb44a5f3c935ad42adc3ee\": container with ID starting with b6db75e823b319ca70be02011a1ea7b9a20ba9e92aeb44a5f3c935ad42adc3ee not found: ID does not exist" Oct 30 01:29:05.419535 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:05.419439 5300 scope.go:122] "RemoveContainer" containerID="970d0565450e21b0ef7693b3afc397a52d15f09b9a33c0954732b2b1f66d1d6a" Oct 30 01:29:05.431822 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:05.431802 5300 scope.go:122] "RemoveContainer" containerID="1a8a522e7b395c6e2a170ad1cfee67b899241e218c1d080eae472f64f7b5527f" Oct 30 01:29:05.444967 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:05.444943 5300 scope.go:122] "RemoveContainer" containerID="970d0565450e21b0ef7693b3afc397a52d15f09b9a33c0954732b2b1f66d1d6a" Oct 30 01:29:05.445552 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:29:05.445332 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"970d0565450e21b0ef7693b3afc397a52d15f09b9a33c0954732b2b1f66d1d6a\": container with ID starting with 970d0565450e21b0ef7693b3afc397a52d15f09b9a33c0954732b2b1f66d1d6a not found: ID does not exist" containerID="970d0565450e21b0ef7693b3afc397a52d15f09b9a33c0954732b2b1f66d1d6a" Oct 30 01:29:05.445552 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:05.445369 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"970d0565450e21b0ef7693b3afc397a52d15f09b9a33c0954732b2b1f66d1d6a"} err="failed to get container status \"970d0565450e21b0ef7693b3afc397a52d15f09b9a33c0954732b2b1f66d1d6a\": rpc error: code = NotFound desc = could not find container \"970d0565450e21b0ef7693b3afc397a52d15f09b9a33c0954732b2b1f66d1d6a\": container with ID starting with 970d0565450e21b0ef7693b3afc397a52d15f09b9a33c0954732b2b1f66d1d6a not found: ID does not exist" Oct 30 01:29:05.445552 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:05.445394 5300 scope.go:122] "RemoveContainer" containerID="1a8a522e7b395c6e2a170ad1cfee67b899241e218c1d080eae472f64f7b5527f" Oct 30 01:29:05.446255 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:29:05.446175 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a8a522e7b395c6e2a170ad1cfee67b899241e218c1d080eae472f64f7b5527f\": container with ID starting with 1a8a522e7b395c6e2a170ad1cfee67b899241e218c1d080eae472f64f7b5527f not found: ID does not exist" containerID="1a8a522e7b395c6e2a170ad1cfee67b899241e218c1d080eae472f64f7b5527f" Oct 30 01:29:05.446255 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:05.446202 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a8a522e7b395c6e2a170ad1cfee67b899241e218c1d080eae472f64f7b5527f"} err="failed to get container status \"1a8a522e7b395c6e2a170ad1cfee67b899241e218c1d080eae472f64f7b5527f\": rpc error: code = NotFound desc = could not find container \"1a8a522e7b395c6e2a170ad1cfee67b899241e218c1d080eae472f64f7b5527f\": container with ID starting with 1a8a522e7b395c6e2a170ad1cfee67b899241e218c1d080eae472f64f7b5527f not found: ID does not exist" Oct 30 01:29:05.451276 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:05.451253 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp"] Oct 30 01:29:05.457194 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:05.457169 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e2d6d3-openshift-default-6f8596d44c-ddkdp"] Oct 30 01:29:05.514609 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:05.514579 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp"] Oct 30 01:29:05.518083 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:05.518063 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-262230-openshift-default-9d45d8bbc-2hptp"] Oct 30 01:29:05.823264 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:05.823232 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1455aae2-7d1f-4788-9a94-7469bee17334" path="/var/lib/kubelet/pods/1455aae2-7d1f-4788-9a94-7469bee17334/volumes" Oct 30 01:29:05.823743 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:05.823729 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce797d22-c33f-4853-8c58-e014cd26bf5b" path="/var/lib/kubelet/pods/ce797d22-c33f-4853-8c58-e014cd26bf5b/volumes" Oct 30 01:29:05.824226 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:05.824210 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d13883c2-ec67-4526-985d-f22c6673853f" path="/var/lib/kubelet/pods/d13883c2-ec67-4526-985d-f22c6673853f/volumes" Oct 30 01:29:06.189200 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:06.189121 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:06.289574 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:06.289542 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:06.886017 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:06.885975 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:08.446139 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:08.446098 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:10.132036 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:10.131996 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:10.573005 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:10.572577 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:10.671100 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:10.671063 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:11.314116 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:11.314085 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4"] Oct 30 01:29:11.356479 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:11.356447 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw"] Oct 30 01:29:11.457575 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:11.457541 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:11.495195 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:11.495160 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8"] Oct 30 01:29:11.495453 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:11.495427 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8" podUID="5f37290a-2d88-43bc-8af1-64bff456c157" containerName="maas-api" containerID="cri-o://d8427b2be59f630dd354ef1d8e83779122890618f547dc1401ea06ffc14c8486" gracePeriod=30 Oct 30 01:29:11.561543 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:11.561503 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:11.896287 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:11.896259 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8" Oct 30 01:29:11.902725 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:11.900455 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f37290a-2d88-43bc-8af1-64bff456c157-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f37290a-2d88-43bc-8af1-64bff456c157-maas-api-tls\") pod \"5f37290a-2d88-43bc-8af1-64bff456c157\" (UID: \"5f37290a-2d88-43bc-8af1-64bff456c157\") " Oct 30 01:29:11.902725 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:11.900679 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f37290a-2d88-43bc-8af1-64bff456c157-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f37290a-2d88-43bc-8af1-64bff456c157-openshift-service-ca\") pod \"5f37290a-2d88-43bc-8af1-64bff456c157\" (UID: \"5f37290a-2d88-43bc-8af1-64bff456c157\") " Oct 30 01:29:11.902725 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:11.900722 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f37290a-2d88-43bc-8af1-64bff456c157-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f37290a-2d88-43bc-8af1-64bff456c157-metrics-certs\") pod \"5f37290a-2d88-43bc-8af1-64bff456c157\" (UID: \"5f37290a-2d88-43bc-8af1-64bff456c157\") " Oct 30 01:29:11.902725 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:11.900843 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f37290a-2d88-43bc-8af1-64bff456c157-kube-api-access-dzkzh\" (UniqueName: \"kubernetes.io/projected/5f37290a-2d88-43bc-8af1-64bff456c157-kube-api-access-dzkzh\") pod \"5f37290a-2d88-43bc-8af1-64bff456c157\" (UID: \"5f37290a-2d88-43bc-8af1-64bff456c157\") " Oct 30 01:29:11.902725 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:11.902720 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f37290a-2d88-43bc-8af1-64bff456c157-openshift-service-ca" pod "5f37290a-2d88-43bc-8af1-64bff456c157" (UID: "5f37290a-2d88-43bc-8af1-64bff456c157"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:11.904039 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:11.904017 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f37290a-2d88-43bc-8af1-64bff456c157-kube-api-access-dzkzh" pod "5f37290a-2d88-43bc-8af1-64bff456c157" (UID: "5f37290a-2d88-43bc-8af1-64bff456c157"). InnerVolumeSpecName "kube-api-access-dzkzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:11.904168 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:11.904096 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f37290a-2d88-43bc-8af1-64bff456c157-maas-api-tls" pod "5f37290a-2d88-43bc-8af1-64bff456c157" (UID: "5f37290a-2d88-43bc-8af1-64bff456c157"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:11.905166 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:11.905141 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f37290a-2d88-43bc-8af1-64bff456c157-metrics-certs" pod "5f37290a-2d88-43bc-8af1-64bff456c157" (UID: "5f37290a-2d88-43bc-8af1-64bff456c157"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:12.009785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:12.003061 5300 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f37290a-2d88-43bc-8af1-64bff456c157-openshift-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:12.009785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:12.003103 5300 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f37290a-2d88-43bc-8af1-64bff456c157-metrics-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:12.009785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:12.003118 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzkzh\" (UniqueName: \"kubernetes.io/projected/5f37290a-2d88-43bc-8af1-64bff456c157-kube-api-access-dzkzh\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:12.009785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:12.003131 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f37290a-2d88-43bc-8af1-64bff456c157-maas-api-tls\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:12.040081 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:12.039976 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:12.165528 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:12.165437 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:12.371899 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:12.371862 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:12.449220 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:12.449132 5300 generic.go:356] "Generic (PLEG): container finished" podID="5f37290a-2d88-43bc-8af1-64bff456c157" containerID="d8427b2be59f630dd354ef1d8e83779122890618f547dc1401ea06ffc14c8486" exitCode=0 Oct 30 01:29:12.449220 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:12.449186 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8" Oct 30 01:29:12.449419 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:12.449212 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8" event={"ID":"5f37290a-2d88-43bc-8af1-64bff456c157","Type":"ContainerDied","Data":"d8427b2be59f630dd354ef1d8e83779122890618f547dc1401ea06ffc14c8486"} Oct 30 01:29:12.449419 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:12.449286 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8" event={"ID":"5f37290a-2d88-43bc-8af1-64bff456c157","Type":"ContainerDied","Data":"db53d71be02ab00b4d4b7838e106c6f2794ea03331bffbcf602f11d2bf54b7f4"} Oct 30 01:29:12.449419 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:12.449321 5300 scope.go:122] "RemoveContainer" containerID="d8427b2be59f630dd354ef1d8e83779122890618f547dc1401ea06ffc14c8486" Oct 30 01:29:12.461606 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:12.461582 5300 scope.go:122] "RemoveContainer" containerID="d8427b2be59f630dd354ef1d8e83779122890618f547dc1401ea06ffc14c8486" Oct 30 01:29:12.461959 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:29:12.461936 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8427b2be59f630dd354ef1d8e83779122890618f547dc1401ea06ffc14c8486\": container with ID starting with d8427b2be59f630dd354ef1d8e83779122890618f547dc1401ea06ffc14c8486 not found: ID does not exist" containerID="d8427b2be59f630dd354ef1d8e83779122890618f547dc1401ea06ffc14c8486" Oct 30 01:29:12.462058 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:12.461969 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8427b2be59f630dd354ef1d8e83779122890618f547dc1401ea06ffc14c8486"} err="failed to get container status \"d8427b2be59f630dd354ef1d8e83779122890618f547dc1401ea06ffc14c8486\": rpc error: code = NotFound desc = could not find container \"d8427b2be59f630dd354ef1d8e83779122890618f547dc1401ea06ffc14c8486\": container with ID starting with d8427b2be59f630dd354ef1d8e83779122890618f547dc1401ea06ffc14c8486 not found: ID does not exist" Oct 30 01:29:12.508777 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:12.508725 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8"] Oct 30 01:29:12.521935 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:12.521905 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8"] Oct 30 01:29:12.673171 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:12.673138 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:13.822752 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:13.822719 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f37290a-2d88-43bc-8af1-64bff456c157" path="/var/lib/kubelet/pods/5f37290a-2d88-43bc-8af1-64bff456c157/volumes" Oct 30 01:29:13.962596 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:13.962556 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:14.645811 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:14.645742 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:14.743474 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:14.743443 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:14.876796 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:14.876732 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2be695-57c48dd7f5-bfxg8" podUID="5f37290a-2d88-43bc-8af1-64bff456c157" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.92:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:29:15.592199 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:15.592162 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:16.314634 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:16.314591 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4" podUID="8af3eb07-ad1d-4484-a586-5a3f71386369" containerName="payload-processing" containerID="cri-o://dc0aad0bb30f7ca076b32b0d380d8eb1468cbb3a400d7da1b61790b3b7cecfdc" gracePeriod=25 Oct 30 01:29:16.356972 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:16.356932 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw" podUID="c0c76374-4140-42c2-9a0d-48a94a0e04b7" containerName="payload-pre-processing" containerID="cri-o://0b4e588f70888c3c55e3f54b6246afb20e4073590ae6c8f360eedf09dec0081b" gracePeriod=25 Oct 30 01:29:17.741820 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:17.741744 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:18.465225 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:18.465192 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:18.605443 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:18.605405 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:18.808374 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:18.808333 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5"] Oct 30 01:29:18.808860 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:18.808640 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" podUID="653f0ddb-957e-4dae-85c9-b5822892df36" containerName="istio-proxy" containerID="cri-o://606ebe0aff4314afb463b0fc678e264041553c36c6bef8ae5bc95637b2cabbdb" gracePeriod=30 Oct 30 01:29:18.814200 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:18.814170 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v"] Oct 30 01:29:18.814386 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:18.814362 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" podUID="86e79fad-5a98-4840-9218-41ddff154453" containerName="istio-proxy" containerID="cri-o://f34bf18efdb30601371cf1f26c192b11529768258198d42218262f19ef534a91" gracePeriod=30 Oct 30 01:29:18.821848 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:18.821826 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx"] Oct 30 01:29:18.822054 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:18.822033 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" podUID="b03e0096-ff6f-4e17-b213-72a40602bf18" containerName="istio-proxy" containerID="cri-o://3c3ad4a44e7fedf7e8ba15ed1fae3b7be55b57b295a747527c69ea56d5a017e6" gracePeriod=30 Oct 30 01:29:18.826870 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:18.826784 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:19.249226 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:19.249194 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:19.352606 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:19.352568 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:19.366884 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:19.366846 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4" podUID="8af3eb07-ad1d-4484-a586-5a3f71386369" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:29:19.367057 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:19.366943 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw" podUID="c0c76374-4140-42c2-9a0d-48a94a0e04b7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:29:20.603630 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:20.603593 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4"] Oct 30 01:29:20.604160 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:20.603842 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4" podUID="26ccc4a1-e4a8-492b-abef-87dd46d9dab3" containerName="maas-api" containerID="cri-o://9a92b44b3beee98180f9f5a0651402d10b1933d1c054881ad9a91acd1edd8155" gracePeriod=30 Oct 30 01:29:20.659910 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:20.659880 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:20.941963 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:20.941936 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4" Oct 30 01:29:21.126475 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:21.117781 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-metrics-certs\") pod \"26ccc4a1-e4a8-492b-abef-87dd46d9dab3\" (UID: \"26ccc4a1-e4a8-492b-abef-87dd46d9dab3\") " Oct 30 01:29:21.126475 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:21.118099 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-maas-api-tls\") pod \"26ccc4a1-e4a8-492b-abef-87dd46d9dab3\" (UID: \"26ccc4a1-e4a8-492b-abef-87dd46d9dab3\") " Oct 30 01:29:21.126475 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:21.118152 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-openshift-service-ca\") pod \"26ccc4a1-e4a8-492b-abef-87dd46d9dab3\" (UID: \"26ccc4a1-e4a8-492b-abef-87dd46d9dab3\") " Oct 30 01:29:21.126475 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:21.118221 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-kube-api-access-rzfww\" (UniqueName: \"kubernetes.io/projected/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-kube-api-access-rzfww\") pod \"26ccc4a1-e4a8-492b-abef-87dd46d9dab3\" (UID: \"26ccc4a1-e4a8-492b-abef-87dd46d9dab3\") " Oct 30 01:29:21.126475 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:21.123449 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-openshift-service-ca" pod "26ccc4a1-e4a8-492b-abef-87dd46d9dab3" (UID: "26ccc4a1-e4a8-492b-abef-87dd46d9dab3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:21.159787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:21.152278 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-kube-api-access-rzfww" pod "26ccc4a1-e4a8-492b-abef-87dd46d9dab3" (UID: "26ccc4a1-e4a8-492b-abef-87dd46d9dab3"). InnerVolumeSpecName "kube-api-access-rzfww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:21.159787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:21.155712 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-maas-api-tls" pod "26ccc4a1-e4a8-492b-abef-87dd46d9dab3" (UID: "26ccc4a1-e4a8-492b-abef-87dd46d9dab3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:21.159787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:21.156501 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-metrics-certs" pod "26ccc4a1-e4a8-492b-abef-87dd46d9dab3" (UID: "26ccc4a1-e4a8-492b-abef-87dd46d9dab3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:21.221346 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:21.221309 5300 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-metrics-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:21.221346 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:21.221340 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-maas-api-tls\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:21.221346 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:21.221349 5300 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-openshift-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:21.221346 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:21.221359 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzfww\" (UniqueName: \"kubernetes.io/projected/26ccc4a1-e4a8-492b-abef-87dd46d9dab3-kube-api-access-rzfww\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:21.353593 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:21.353547 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:21.462561 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:21.462474 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:21.532419 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:21.532383 5300 generic.go:356] "Generic (PLEG): container finished" podID="26ccc4a1-e4a8-492b-abef-87dd46d9dab3" containerID="9a92b44b3beee98180f9f5a0651402d10b1933d1c054881ad9a91acd1edd8155" exitCode=0 Oct 30 01:29:21.532621 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:21.532435 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4" Oct 30 01:29:21.532621 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:21.532475 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4" event={"ID":"26ccc4a1-e4a8-492b-abef-87dd46d9dab3","Type":"ContainerDied","Data":"9a92b44b3beee98180f9f5a0651402d10b1933d1c054881ad9a91acd1edd8155"} Oct 30 01:29:21.532621 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:21.532534 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4" event={"ID":"26ccc4a1-e4a8-492b-abef-87dd46d9dab3","Type":"ContainerDied","Data":"14287288fdecad25a6c2b3df47f4007a7ec0a70d002bf7c2278b2a629b606fbf"} Oct 30 01:29:21.532621 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:21.532563 5300 scope.go:122] "RemoveContainer" containerID="9a92b44b3beee98180f9f5a0651402d10b1933d1c054881ad9a91acd1edd8155" Oct 30 01:29:21.543574 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:21.543557 5300 scope.go:122] "RemoveContainer" containerID="9a92b44b3beee98180f9f5a0651402d10b1933d1c054881ad9a91acd1edd8155" Oct 30 01:29:21.543856 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:29:21.543836 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a92b44b3beee98180f9f5a0651402d10b1933d1c054881ad9a91acd1edd8155\": container with ID starting with 9a92b44b3beee98180f9f5a0651402d10b1933d1c054881ad9a91acd1edd8155 not found: ID does not exist" containerID="9a92b44b3beee98180f9f5a0651402d10b1933d1c054881ad9a91acd1edd8155" Oct 30 01:29:21.543904 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:21.543865 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a92b44b3beee98180f9f5a0651402d10b1933d1c054881ad9a91acd1edd8155"} err="failed to get container status \"9a92b44b3beee98180f9f5a0651402d10b1933d1c054881ad9a91acd1edd8155\": rpc error: code = NotFound desc = could not find container \"9a92b44b3beee98180f9f5a0651402d10b1933d1c054881ad9a91acd1edd8155\": container with ID starting with 9a92b44b3beee98180f9f5a0651402d10b1933d1c054881ad9a91acd1edd8155 not found: ID does not exist" Oct 30 01:29:21.551624 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:21.551606 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:21.597109 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:21.597074 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4"] Oct 30 01:29:21.602660 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:21.602615 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-64f472-648c7c4b85-cl7s4"] Oct 30 01:29:21.628448 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:29:21.628408 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:29:21.822442 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:21.822408 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26ccc4a1-e4a8-492b-abef-87dd46d9dab3" path="/var/lib/kubelet/pods/26ccc4a1-e4a8-492b-abef-87dd46d9dab3/volumes" Oct 30 01:29:21.983296 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:21.983249 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:23.366530 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:23.366488 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:24.121871 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.121841 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:29:24.137533 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.137507 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:29:24.149411 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.149387 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:29:24.161217 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.161189 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86e79fad-5a98-4840-9218-41ddff154453-kube-api-access-x75pz\" (UniqueName: \"kubernetes.io/projected/86e79fad-5a98-4840-9218-41ddff154453-kube-api-access-x75pz\") pod \"86e79fad-5a98-4840-9218-41ddff154453\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " Oct 30 01:29:24.161379 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.161358 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-workload-socket\") pod \"653f0ddb-957e-4dae-85c9-b5822892df36\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " Oct 30 01:29:24.161482 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.161467 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-istio-envoy\") pod \"653f0ddb-957e-4dae-85c9-b5822892df36\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " Oct 30 01:29:24.161534 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.161507 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/653f0ddb-957e-4dae-85c9-b5822892df36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/653f0ddb-957e-4dae-85c9-b5822892df36-istio-podinfo\") pod \"653f0ddb-957e-4dae-85c9-b5822892df36\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " Oct 30 01:29:24.161601 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.161585 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/653f0ddb-957e-4dae-85c9-b5822892df36-istio-token\" (UniqueName: \"kubernetes.io/projected/653f0ddb-957e-4dae-85c9-b5822892df36-istio-token\") pod \"653f0ddb-957e-4dae-85c9-b5822892df36\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " Oct 30 01:29:24.161654 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.161619 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-credential-socket\") pod \"86e79fad-5a98-4840-9218-41ddff154453\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " Oct 30 01:29:24.161709 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.161660 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86e79fad-5a98-4840-9218-41ddff154453-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e79fad-5a98-4840-9218-41ddff154453-istiod-ca-cert\") pod \"86e79fad-5a98-4840-9218-41ddff154453\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " Oct 30 01:29:24.161709 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.161691 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/653f0ddb-957e-4dae-85c9-b5822892df36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/653f0ddb-957e-4dae-85c9-b5822892df36-istiod-ca-cert\") pod \"653f0ddb-957e-4dae-85c9-b5822892df36\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " Oct 30 01:29:24.161834 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.161745 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86e79fad-5a98-4840-9218-41ddff154453-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e79fad-5a98-4840-9218-41ddff154453-istio-podinfo\") pod \"86e79fad-5a98-4840-9218-41ddff154453\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " Oct 30 01:29:24.161834 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.161787 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-credential-socket\") pod \"653f0ddb-957e-4dae-85c9-b5822892df36\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " Oct 30 01:29:24.161955 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.161833 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-istio-envoy\") pod \"86e79fad-5a98-4840-9218-41ddff154453\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " Oct 30 01:29:24.161955 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.161894 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/653f0ddb-957e-4dae-85c9-b5822892df36-kube-api-access-mtthd\" (UniqueName: \"kubernetes.io/projected/653f0ddb-957e-4dae-85c9-b5822892df36-kube-api-access-mtthd\") pod \"653f0ddb-957e-4dae-85c9-b5822892df36\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " Oct 30 01:29:24.161955 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.161933 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-workload-certs\") pod \"653f0ddb-957e-4dae-85c9-b5822892df36\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " Oct 30 01:29:24.162234 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.162202 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-workload-certs\") pod \"86e79fad-5a98-4840-9218-41ddff154453\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " Oct 30 01:29:24.162353 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.162341 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-istio-data\") pod \"653f0ddb-957e-4dae-85c9-b5822892df36\" (UID: \"653f0ddb-957e-4dae-85c9-b5822892df36\") " Oct 30 01:29:24.162423 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.161831 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-workload-socket" pod "653f0ddb-957e-4dae-85c9-b5822892df36" (UID: "653f0ddb-957e-4dae-85c9-b5822892df36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.162423 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.162107 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86e79fad-5a98-4840-9218-41ddff154453-istiod-ca-cert" pod "86e79fad-5a98-4840-9218-41ddff154453" (UID: "86e79fad-5a98-4840-9218-41ddff154453"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:24.162423 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.162378 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-credential-socket" pod "86e79fad-5a98-4840-9218-41ddff154453" (UID: "86e79fad-5a98-4840-9218-41ddff154453"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.162598 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.162480 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-istio-data\") pod \"86e79fad-5a98-4840-9218-41ddff154453\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " Oct 30 01:29:24.162598 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.162545 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86e79fad-5a98-4840-9218-41ddff154453-istio-token\" (UniqueName: \"kubernetes.io/projected/86e79fad-5a98-4840-9218-41ddff154453-istio-token\") pod \"86e79fad-5a98-4840-9218-41ddff154453\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " Oct 30 01:29:24.162717 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.162602 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-workload-certs" pod "86e79fad-5a98-4840-9218-41ddff154453" (UID: "86e79fad-5a98-4840-9218-41ddff154453"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.162717 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.162644 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-workload-socket\") pod \"86e79fad-5a98-4840-9218-41ddff154453\" (UID: \"86e79fad-5a98-4840-9218-41ddff154453\") " Oct 30 01:29:24.162717 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.162674 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/653f0ddb-957e-4dae-85c9-b5822892df36-istiod-ca-cert" pod "653f0ddb-957e-4dae-85c9-b5822892df36" (UID: "653f0ddb-957e-4dae-85c9-b5822892df36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:24.162933 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.162811 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-workload-certs" pod "653f0ddb-957e-4dae-85c9-b5822892df36" (UID: "653f0ddb-957e-4dae-85c9-b5822892df36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.162933 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.162840 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-istio-data" pod "653f0ddb-957e-4dae-85c9-b5822892df36" (UID: "653f0ddb-957e-4dae-85c9-b5822892df36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.163055 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.162956 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-credential-socket" pod "653f0ddb-957e-4dae-85c9-b5822892df36" (UID: "653f0ddb-957e-4dae-85c9-b5822892df36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.163121 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.163056 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-istio-data" pod "86e79fad-5a98-4840-9218-41ddff154453" (UID: "86e79fad-5a98-4840-9218-41ddff154453"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.163121 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.163069 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-workload-socket" pod "86e79fad-5a98-4840-9218-41ddff154453" (UID: "86e79fad-5a98-4840-9218-41ddff154453"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.163731 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.163704 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.163910 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.163737 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.163910 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.163755 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e79fad-5a98-4840-9218-41ddff154453-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.163910 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.163791 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/653f0ddb-957e-4dae-85c9-b5822892df36-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.163910 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.163806 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.163910 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.163820 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.163910 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.163835 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.163910 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.163848 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.163910 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.163862 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.163910 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.163876 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.164560 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.164369 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-istio-envoy" pod "86e79fad-5a98-4840-9218-41ddff154453" (UID: "86e79fad-5a98-4840-9218-41ddff154453"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.164676 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.164646 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e79fad-5a98-4840-9218-41ddff154453-kube-api-access-x75pz" pod "86e79fad-5a98-4840-9218-41ddff154453" (UID: "86e79fad-5a98-4840-9218-41ddff154453"). InnerVolumeSpecName "kube-api-access-x75pz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:24.164819 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.164802 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-istio-envoy" pod "653f0ddb-957e-4dae-85c9-b5822892df36" (UID: "653f0ddb-957e-4dae-85c9-b5822892df36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.164987 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.164968 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86e79fad-5a98-4840-9218-41ddff154453-istio-podinfo" pod "86e79fad-5a98-4840-9218-41ddff154453" (UID: "86e79fad-5a98-4840-9218-41ddff154453"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:29:24.165240 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.165214 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653f0ddb-957e-4dae-85c9-b5822892df36-istio-token" pod "653f0ddb-957e-4dae-85c9-b5822892df36" (UID: "653f0ddb-957e-4dae-85c9-b5822892df36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:24.165411 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.165393 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e79fad-5a98-4840-9218-41ddff154453-istio-token" pod "86e79fad-5a98-4840-9218-41ddff154453" (UID: "86e79fad-5a98-4840-9218-41ddff154453"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:24.165641 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.165623 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/653f0ddb-957e-4dae-85c9-b5822892df36-istio-podinfo" pod "653f0ddb-957e-4dae-85c9-b5822892df36" (UID: "653f0ddb-957e-4dae-85c9-b5822892df36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:29:24.165866 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.165847 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653f0ddb-957e-4dae-85c9-b5822892df36-kube-api-access-mtthd" pod "653f0ddb-957e-4dae-85c9-b5822892df36" (UID: "653f0ddb-957e-4dae-85c9-b5822892df36"). InnerVolumeSpecName "kube-api-access-mtthd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:24.220132 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.220076 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn"] Oct 30 01:29:24.265117 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.265079 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-workload-socket\") pod \"b03e0096-ff6f-4e17-b213-72a40602bf18\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " Oct 30 01:29:24.265337 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.265159 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b03e0096-ff6f-4e17-b213-72a40602bf18-kube-api-access-4p8ml\" (UniqueName: \"kubernetes.io/projected/b03e0096-ff6f-4e17-b213-72a40602bf18-kube-api-access-4p8ml\") pod \"b03e0096-ff6f-4e17-b213-72a40602bf18\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " Oct 30 01:29:24.265337 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.265324 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-podinfo\") pod \"b03e0096-ff6f-4e17-b213-72a40602bf18\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " Oct 30 01:29:24.265484 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.265386 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-workload-certs\") pod \"b03e0096-ff6f-4e17-b213-72a40602bf18\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " Oct 30 01:29:24.265484 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.265443 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b03e0096-ff6f-4e17-b213-72a40602bf18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b03e0096-ff6f-4e17-b213-72a40602bf18-istiod-ca-cert\") pod \"b03e0096-ff6f-4e17-b213-72a40602bf18\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " Oct 30 01:29:24.265600 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.265550 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-token\" (UniqueName: \"kubernetes.io/projected/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-token\") pod \"b03e0096-ff6f-4e17-b213-72a40602bf18\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " Oct 30 01:29:24.266364 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.265631 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-data\") pod \"b03e0096-ff6f-4e17-b213-72a40602bf18\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " Oct 30 01:29:24.266364 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.265663 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-credential-socket\") pod \"b03e0096-ff6f-4e17-b213-72a40602bf18\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " Oct 30 01:29:24.266364 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.265709 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-envoy\") pod \"b03e0096-ff6f-4e17-b213-72a40602bf18\" (UID: \"b03e0096-ff6f-4e17-b213-72a40602bf18\") " Oct 30 01:29:24.266364 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.265885 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-workload-certs" pod "b03e0096-ff6f-4e17-b213-72a40602bf18" (UID: "b03e0096-ff6f-4e17-b213-72a40602bf18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.266364 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.266334 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-credential-socket" pod "b03e0096-ff6f-4e17-b213-72a40602bf18" (UID: "b03e0096-ff6f-4e17-b213-72a40602bf18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.266364 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.266348 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-data" pod "b03e0096-ff6f-4e17-b213-72a40602bf18" (UID: "b03e0096-ff6f-4e17-b213-72a40602bf18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.266721 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.266424 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b03e0096-ff6f-4e17-b213-72a40602bf18-istiod-ca-cert" pod "b03e0096-ff6f-4e17-b213-72a40602bf18" (UID: "b03e0096-ff6f-4e17-b213-72a40602bf18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:24.266721 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.266486 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.266721 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.266506 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.266721 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.266521 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86e79fad-5a98-4840-9218-41ddff154453-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.266721 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.266537 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x75pz\" (UniqueName: \"kubernetes.io/projected/86e79fad-5a98-4840-9218-41ddff154453-kube-api-access-x75pz\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.266721 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.266552 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/653f0ddb-957e-4dae-85c9-b5822892df36-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.266721 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.266566 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/653f0ddb-957e-4dae-85c9-b5822892df36-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.266721 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.266578 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/653f0ddb-957e-4dae-85c9-b5822892df36-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.266721 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.266591 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e79fad-5a98-4840-9218-41ddff154453-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.266721 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.266605 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e79fad-5a98-4840-9218-41ddff154453-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.266721 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.266617 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.266721 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.266630 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtthd\" (UniqueName: \"kubernetes.io/projected/653f0ddb-957e-4dae-85c9-b5822892df36-kube-api-access-mtthd\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.267377 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.266845 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-workload-socket" pod "b03e0096-ff6f-4e17-b213-72a40602bf18" (UID: "b03e0096-ff6f-4e17-b213-72a40602bf18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.269082 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.269050 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-envoy" pod "b03e0096-ff6f-4e17-b213-72a40602bf18" (UID: "b03e0096-ff6f-4e17-b213-72a40602bf18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:24.272213 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.272186 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03e0096-ff6f-4e17-b213-72a40602bf18-kube-api-access-4p8ml" pod "b03e0096-ff6f-4e17-b213-72a40602bf18" (UID: "b03e0096-ff6f-4e17-b213-72a40602bf18"). InnerVolumeSpecName "kube-api-access-4p8ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:24.272320 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.272285 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-podinfo" pod "b03e0096-ff6f-4e17-b213-72a40602bf18" (UID: "b03e0096-ff6f-4e17-b213-72a40602bf18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:29:24.273864 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.273837 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-token" pod "b03e0096-ff6f-4e17-b213-72a40602bf18" (UID: "b03e0096-ff6f-4e17-b213-72a40602bf18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:24.358095 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.358007 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:24.367679 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.367646 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.367679 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.367675 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b03e0096-ff6f-4e17-b213-72a40602bf18-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.367679 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.367683 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.368175 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.367691 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.368175 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.367701 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b03e0096-ff6f-4e17-b213-72a40602bf18-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.368175 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.367712 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p8ml\" (UniqueName: \"kubernetes.io/projected/b03e0096-ff6f-4e17-b213-72a40602bf18-kube-api-access-4p8ml\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:24.553658 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.553623 5300 generic.go:356] "Generic (PLEG): container finished" podID="653f0ddb-957e-4dae-85c9-b5822892df36" containerID="606ebe0aff4314afb463b0fc678e264041553c36c6bef8ae5bc95637b2cabbdb" exitCode=0 Oct 30 01:29:24.553855 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.553677 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" Oct 30 01:29:24.553855 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.553712 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" event={"ID":"653f0ddb-957e-4dae-85c9-b5822892df36","Type":"ContainerDied","Data":"606ebe0aff4314afb463b0fc678e264041553c36c6bef8ae5bc95637b2cabbdb"} Oct 30 01:29:24.553855 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.553788 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5" event={"ID":"653f0ddb-957e-4dae-85c9-b5822892df36","Type":"ContainerDied","Data":"cae8043b39ee4bf4a486b9ecc04f2d25ddc89f72d015a86b0af35fd2dbe43013"} Oct 30 01:29:24.553855 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.553817 5300 scope.go:122] "RemoveContainer" containerID="606ebe0aff4314afb463b0fc678e264041553c36c6bef8ae5bc95637b2cabbdb" Oct 30 01:29:24.555463 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.555439 5300 generic.go:356] "Generic (PLEG): container finished" podID="b03e0096-ff6f-4e17-b213-72a40602bf18" containerID="3c3ad4a44e7fedf7e8ba15ed1fae3b7be55b57b295a747527c69ea56d5a017e6" exitCode=0 Oct 30 01:29:24.555603 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.555517 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" event={"ID":"b03e0096-ff6f-4e17-b213-72a40602bf18","Type":"ContainerDied","Data":"3c3ad4a44e7fedf7e8ba15ed1fae3b7be55b57b295a747527c69ea56d5a017e6"} Oct 30 01:29:24.555603 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.555580 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" event={"ID":"b03e0096-ff6f-4e17-b213-72a40602bf18","Type":"ContainerDied","Data":"1dacdfd4ae81f26cdbc4af3a5c4422e7e74629dda69b57da4ec921d895bea535"} Oct 30 01:29:24.555603 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.555592 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx" Oct 30 01:29:24.557162 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.557142 5300 generic.go:356] "Generic (PLEG): container finished" podID="86e79fad-5a98-4840-9218-41ddff154453" containerID="f34bf18efdb30601371cf1f26c192b11529768258198d42218262f19ef534a91" exitCode=0 Oct 30 01:29:24.557290 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.557200 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" Oct 30 01:29:24.557290 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.557204 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" event={"ID":"86e79fad-5a98-4840-9218-41ddff154453","Type":"ContainerDied","Data":"f34bf18efdb30601371cf1f26c192b11529768258198d42218262f19ef534a91"} Oct 30 01:29:24.557399 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.557315 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v" event={"ID":"86e79fad-5a98-4840-9218-41ddff154453","Type":"ContainerDied","Data":"1ae9962dcb294c01e6154b96b6356cd17d96c294e6b4b43c44830627ff05366c"} Oct 30 01:29:24.565167 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.565135 5300 scope.go:122] "RemoveContainer" containerID="606ebe0aff4314afb463b0fc678e264041553c36c6bef8ae5bc95637b2cabbdb" Oct 30 01:29:24.565496 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:29:24.565473 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"606ebe0aff4314afb463b0fc678e264041553c36c6bef8ae5bc95637b2cabbdb\": container with ID starting with 606ebe0aff4314afb463b0fc678e264041553c36c6bef8ae5bc95637b2cabbdb not found: ID does not exist" containerID="606ebe0aff4314afb463b0fc678e264041553c36c6bef8ae5bc95637b2cabbdb" Oct 30 01:29:24.565566 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.565503 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"606ebe0aff4314afb463b0fc678e264041553c36c6bef8ae5bc95637b2cabbdb"} err="failed to get container status \"606ebe0aff4314afb463b0fc678e264041553c36c6bef8ae5bc95637b2cabbdb\": rpc error: code = NotFound desc = could not find container \"606ebe0aff4314afb463b0fc678e264041553c36c6bef8ae5bc95637b2cabbdb\": container with ID starting with 606ebe0aff4314afb463b0fc678e264041553c36c6bef8ae5bc95637b2cabbdb not found: ID does not exist" Oct 30 01:29:24.565566 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.565520 5300 scope.go:122] "RemoveContainer" containerID="3c3ad4a44e7fedf7e8ba15ed1fae3b7be55b57b295a747527c69ea56d5a017e6" Oct 30 01:29:24.578068 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.578051 5300 scope.go:122] "RemoveContainer" containerID="3c3ad4a44e7fedf7e8ba15ed1fae3b7be55b57b295a747527c69ea56d5a017e6" Oct 30 01:29:24.578381 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:29:24.578364 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c3ad4a44e7fedf7e8ba15ed1fae3b7be55b57b295a747527c69ea56d5a017e6\": container with ID starting with 3c3ad4a44e7fedf7e8ba15ed1fae3b7be55b57b295a747527c69ea56d5a017e6 not found: ID does not exist" containerID="3c3ad4a44e7fedf7e8ba15ed1fae3b7be55b57b295a747527c69ea56d5a017e6" Oct 30 01:29:24.578459 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.578387 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c3ad4a44e7fedf7e8ba15ed1fae3b7be55b57b295a747527c69ea56d5a017e6"} err="failed to get container status \"3c3ad4a44e7fedf7e8ba15ed1fae3b7be55b57b295a747527c69ea56d5a017e6\": rpc error: code = NotFound desc = could not find container \"3c3ad4a44e7fedf7e8ba15ed1fae3b7be55b57b295a747527c69ea56d5a017e6\": container with ID starting with 3c3ad4a44e7fedf7e8ba15ed1fae3b7be55b57b295a747527c69ea56d5a017e6 not found: ID does not exist" Oct 30 01:29:24.578459 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.578404 5300 scope.go:122] "RemoveContainer" containerID="f34bf18efdb30601371cf1f26c192b11529768258198d42218262f19ef534a91" Oct 30 01:29:24.589651 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.589633 5300 scope.go:122] "RemoveContainer" containerID="f34bf18efdb30601371cf1f26c192b11529768258198d42218262f19ef534a91" Oct 30 01:29:24.590024 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:29:24.589999 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f34bf18efdb30601371cf1f26c192b11529768258198d42218262f19ef534a91\": container with ID starting with f34bf18efdb30601371cf1f26c192b11529768258198d42218262f19ef534a91 not found: ID does not exist" containerID="f34bf18efdb30601371cf1f26c192b11529768258198d42218262f19ef534a91" Oct 30 01:29:24.590101 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.590031 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f34bf18efdb30601371cf1f26c192b11529768258198d42218262f19ef534a91"} err="failed to get container status \"f34bf18efdb30601371cf1f26c192b11529768258198d42218262f19ef534a91\": rpc error: code = NotFound desc = could not find container \"f34bf18efdb30601371cf1f26c192b11529768258198d42218262f19ef534a91\": container with ID starting with f34bf18efdb30601371cf1f26c192b11529768258198d42218262f19ef534a91 not found: ID does not exist" Oct 30 01:29:24.615571 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.615476 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5"] Oct 30 01:29:24.620609 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.620572 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4ngp5"] Oct 30 01:29:24.637368 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.637336 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:24.657772 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.657731 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx"] Oct 30 01:29:24.661431 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.661405 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-sf8cx"] Oct 30 01:29:24.702293 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.702222 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v"] Oct 30 01:29:24.708444 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.708414 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2be695-openshift-default-7c8b685f79-4t85v"] Oct 30 01:29:24.816092 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:24.816060 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:25.675922 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:25.675890 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:25.823292 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:25.823256 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="653f0ddb-957e-4dae-85c9-b5822892df36" path="/var/lib/kubelet/pods/653f0ddb-957e-4dae-85c9-b5822892df36/volumes" Oct 30 01:29:25.823916 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:25.823898 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86e79fad-5a98-4840-9218-41ddff154453" path="/var/lib/kubelet/pods/86e79fad-5a98-4840-9218-41ddff154453/volumes" Oct 30 01:29:25.824357 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:25.824344 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b03e0096-ff6f-4e17-b213-72a40602bf18" path="/var/lib/kubelet/pods/b03e0096-ff6f-4e17-b213-72a40602bf18/volumes" Oct 30 01:29:27.740233 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:27.740201 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:28.342038 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:28.342006 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:28.486937 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:28.486901 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:28.704585 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:28.704498 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:29.366205 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:29.366163 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw" podUID="c0c76374-4140-42c2-9a0d-48a94a0e04b7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:29:29.366631 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:29.366173 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4" podUID="8af3eb07-ad1d-4484-a586-5a3f71386369" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:29:29.594635 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:29.594605 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl"] Oct 30 01:29:29.636604 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:29.636490 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw"] Oct 30 01:29:29.659480 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:29.659449 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:29.760229 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:29.760198 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:29.861893 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:29.861847 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:31.373173 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:31.373130 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:33.130934 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:33.130900 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:34.595128 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:34.595084 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl" podUID="d3fb7228-7fdb-4db8-a878-d6f902cbc170" containerName="payload-processing" containerID="cri-o://5af672de892d6565734d950bcf7e7423cdd4122c455ba611690e459ab2e35941" gracePeriod=25 Oct 30 01:29:34.636976 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:34.636938 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw" podUID="1fd8605b-ff02-4868-a951-8fe172766e88" containerName="payload-pre-processing" containerID="cri-o://a0e810e2c3bfc7ea003835e85ce6cdcae08cf85bb01a4d72a4df1df7e5b1afc3" gracePeriod=25 Oct 30 01:29:34.707607 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:34.707576 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:35.306205 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:35.306168 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:36.033485 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.033441 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf"] Oct 30 01:29:36.034045 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.033728 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" podUID="49bcda0b-9448-4e57-8328-220ee6208164" containerName="maas-api" containerID="cri-o://decfd3cf0ee4eb4679b1d625b20e5c8f602c18503495394647387a4b07830fc1" gracePeriod=30 Oct 30 01:29:36.090435 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.090399 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:36.547892 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.547513 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" Oct 30 01:29:36.560152 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.559960 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd"] Oct 30 01:29:36.560633 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.560578 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd" podUID="127ad507-8fcc-4c89-be18-27f08b405f8d" containerName="maas-api" containerID="cri-o://579f38bd1fffebcda0bb22c366c83b2150c2675762942929d16786a9bfd17799" gracePeriod=30 Oct 30 01:29:36.665583 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.662370 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:36.693819 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.686460 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49bcda0b-9448-4e57-8328-220ee6208164-metrics-certs\" (UniqueName: \"kubernetes.io/secret/49bcda0b-9448-4e57-8328-220ee6208164-metrics-certs\") pod \"49bcda0b-9448-4e57-8328-220ee6208164\" (UID: \"49bcda0b-9448-4e57-8328-220ee6208164\") " Oct 30 01:29:36.693819 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.686529 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49bcda0b-9448-4e57-8328-220ee6208164-kube-api-access-hcvck\" (UniqueName: \"kubernetes.io/projected/49bcda0b-9448-4e57-8328-220ee6208164-kube-api-access-hcvck\") pod \"49bcda0b-9448-4e57-8328-220ee6208164\" (UID: \"49bcda0b-9448-4e57-8328-220ee6208164\") " Oct 30 01:29:36.693819 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.686630 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49bcda0b-9448-4e57-8328-220ee6208164-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49bcda0b-9448-4e57-8328-220ee6208164-openshift-service-ca\") pod \"49bcda0b-9448-4e57-8328-220ee6208164\" (UID: \"49bcda0b-9448-4e57-8328-220ee6208164\") " Oct 30 01:29:36.693819 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.686676 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49bcda0b-9448-4e57-8328-220ee6208164-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49bcda0b-9448-4e57-8328-220ee6208164-maas-api-tls\") pod \"49bcda0b-9448-4e57-8328-220ee6208164\" (UID: \"49bcda0b-9448-4e57-8328-220ee6208164\") " Oct 30 01:29:36.693819 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.692812 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49bcda0b-9448-4e57-8328-220ee6208164-openshift-service-ca" pod "49bcda0b-9448-4e57-8328-220ee6208164" (UID: "49bcda0b-9448-4e57-8328-220ee6208164"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:36.701557 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.701514 5300 generic.go:356] "Generic (PLEG): container finished" podID="c0c76374-4140-42c2-9a0d-48a94a0e04b7" containerID="0b4e588f70888c3c55e3f54b6246afb20e4073590ae6c8f360eedf09dec0081b" exitCode=0 Oct 30 01:29:36.701731 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.701634 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw" event={"ID":"c0c76374-4140-42c2-9a0d-48a94a0e04b7","Type":"ContainerDied","Data":"0b4e588f70888c3c55e3f54b6246afb20e4073590ae6c8f360eedf09dec0081b"} Oct 30 01:29:36.704229 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.704196 5300 generic.go:356] "Generic (PLEG): container finished" podID="8af3eb07-ad1d-4484-a586-5a3f71386369" containerID="dc0aad0bb30f7ca076b32b0d380d8eb1468cbb3a400d7da1b61790b3b7cecfdc" exitCode=0 Oct 30 01:29:36.704550 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.704524 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4" event={"ID":"8af3eb07-ad1d-4484-a586-5a3f71386369","Type":"ContainerDied","Data":"dc0aad0bb30f7ca076b32b0d380d8eb1468cbb3a400d7da1b61790b3b7cecfdc"} Oct 30 01:29:36.709722 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.708802 5300 generic.go:356] "Generic (PLEG): container finished" podID="49bcda0b-9448-4e57-8328-220ee6208164" containerID="decfd3cf0ee4eb4679b1d625b20e5c8f602c18503495394647387a4b07830fc1" exitCode=0 Oct 30 01:29:36.709722 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.708863 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" event={"ID":"49bcda0b-9448-4e57-8328-220ee6208164","Type":"ContainerDied","Data":"decfd3cf0ee4eb4679b1d625b20e5c8f602c18503495394647387a4b07830fc1"} Oct 30 01:29:36.709722 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.708909 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" event={"ID":"49bcda0b-9448-4e57-8328-220ee6208164","Type":"ContainerDied","Data":"4097b44f7a4d37a0d7f27d11918289bdc93146a8706e697806b153da1080aab5"} Oct 30 01:29:36.709722 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.708942 5300 scope.go:122] "RemoveContainer" containerID="decfd3cf0ee4eb4679b1d625b20e5c8f602c18503495394647387a4b07830fc1" Oct 30 01:29:36.709722 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.709115 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf" Oct 30 01:29:36.709722 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.709619 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49bcda0b-9448-4e57-8328-220ee6208164-metrics-certs" pod "49bcda0b-9448-4e57-8328-220ee6208164" (UID: "49bcda0b-9448-4e57-8328-220ee6208164"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:36.711688 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.710285 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49bcda0b-9448-4e57-8328-220ee6208164-maas-api-tls" pod "49bcda0b-9448-4e57-8328-220ee6208164" (UID: "49bcda0b-9448-4e57-8328-220ee6208164"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:36.748139 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.746664 5300 scope.go:122] "RemoveContainer" containerID="decfd3cf0ee4eb4679b1d625b20e5c8f602c18503495394647387a4b07830fc1" Oct 30 01:29:36.750955 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.748978 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49bcda0b-9448-4e57-8328-220ee6208164-kube-api-access-hcvck" pod "49bcda0b-9448-4e57-8328-220ee6208164" (UID: "49bcda0b-9448-4e57-8328-220ee6208164"). InnerVolumeSpecName "kube-api-access-hcvck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:36.751235 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:29:36.751037 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"decfd3cf0ee4eb4679b1d625b20e5c8f602c18503495394647387a4b07830fc1\": container with ID starting with decfd3cf0ee4eb4679b1d625b20e5c8f602c18503495394647387a4b07830fc1 not found: ID does not exist" containerID="decfd3cf0ee4eb4679b1d625b20e5c8f602c18503495394647387a4b07830fc1" Oct 30 01:29:36.751235 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.751084 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"decfd3cf0ee4eb4679b1d625b20e5c8f602c18503495394647387a4b07830fc1"} err="failed to get container status \"decfd3cf0ee4eb4679b1d625b20e5c8f602c18503495394647387a4b07830fc1\": rpc error: code = NotFound desc = could not find container \"decfd3cf0ee4eb4679b1d625b20e5c8f602c18503495394647387a4b07830fc1\": container with ID starting with decfd3cf0ee4eb4679b1d625b20e5c8f602c18503495394647387a4b07830fc1 not found: ID does not exist" Oct 30 01:29:36.792422 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.792383 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49bcda0b-9448-4e57-8328-220ee6208164-maas-api-tls\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:36.792422 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.792430 5300 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49bcda0b-9448-4e57-8328-220ee6208164-metrics-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:36.792688 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.792444 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcvck\" (UniqueName: \"kubernetes.io/projected/49bcda0b-9448-4e57-8328-220ee6208164-kube-api-access-hcvck\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:36.792688 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.792461 5300 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49bcda0b-9448-4e57-8328-220ee6208164-openshift-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:36.797095 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.796953 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:36.831108 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.831005 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4" Oct 30 01:29:36.914791 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.913894 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8af3eb07-ad1d-4484-a586-5a3f71386369-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8af3eb07-ad1d-4484-a586-5a3f71386369-plugins-config-volume\") pod \"8af3eb07-ad1d-4484-a586-5a3f71386369\" (UID: \"8af3eb07-ad1d-4484-a586-5a3f71386369\") " Oct 30 01:29:36.914791 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.914008 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8af3eb07-ad1d-4484-a586-5a3f71386369-kube-api-access-bsfv9\" (UniqueName: \"kubernetes.io/projected/8af3eb07-ad1d-4484-a586-5a3f71386369-kube-api-access-bsfv9\") pod \"8af3eb07-ad1d-4484-a586-5a3f71386369\" (UID: \"8af3eb07-ad1d-4484-a586-5a3f71386369\") " Oct 30 01:29:36.932913 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.916298 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8af3eb07-ad1d-4484-a586-5a3f71386369-plugins-config-volume" pod "8af3eb07-ad1d-4484-a586-5a3f71386369" (UID: "8af3eb07-ad1d-4484-a586-5a3f71386369"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:36.932913 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.920807 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8af3eb07-ad1d-4484-a586-5a3f71386369-kube-api-access-bsfv9" pod "8af3eb07-ad1d-4484-a586-5a3f71386369" (UID: "8af3eb07-ad1d-4484-a586-5a3f71386369"). InnerVolumeSpecName "kube-api-access-bsfv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:36.932913 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.929043 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:36.999719 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:36.999692 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw" Oct 30 01:29:37.019726 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.019696 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8af3eb07-ad1d-4484-a586-5a3f71386369-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:37.019726 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.019722 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsfv9\" (UniqueName: \"kubernetes.io/projected/8af3eb07-ad1d-4484-a586-5a3f71386369-kube-api-access-bsfv9\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:37.062850 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.061717 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd" Oct 30 01:29:37.142898 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.140836 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf"] Oct 30 01:29:37.142898 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.142319 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-999ff9-5f7588c956-l4qmf"] Oct 30 01:29:37.157287 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.157241 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/127ad507-8fcc-4c89-be18-27f08b405f8d-kube-api-access-k77xh\" (UniqueName: \"kubernetes.io/projected/127ad507-8fcc-4c89-be18-27f08b405f8d-kube-api-access-k77xh\") pod \"127ad507-8fcc-4c89-be18-27f08b405f8d\" (UID: \"127ad507-8fcc-4c89-be18-27f08b405f8d\") " Oct 30 01:29:37.157496 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.157331 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/127ad507-8fcc-4c89-be18-27f08b405f8d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/127ad507-8fcc-4c89-be18-27f08b405f8d-metrics-certs\") pod \"127ad507-8fcc-4c89-be18-27f08b405f8d\" (UID: \"127ad507-8fcc-4c89-be18-27f08b405f8d\") " Oct 30 01:29:37.157572 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.157500 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/127ad507-8fcc-4c89-be18-27f08b405f8d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/127ad507-8fcc-4c89-be18-27f08b405f8d-openshift-service-ca\") pod \"127ad507-8fcc-4c89-be18-27f08b405f8d\" (UID: \"127ad507-8fcc-4c89-be18-27f08b405f8d\") " Oct 30 01:29:37.157702 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.157584 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0c76374-4140-42c2-9a0d-48a94a0e04b7-kube-api-access-ls25m\" (UniqueName: \"kubernetes.io/projected/c0c76374-4140-42c2-9a0d-48a94a0e04b7-kube-api-access-ls25m\") pod \"c0c76374-4140-42c2-9a0d-48a94a0e04b7\" (UID: \"c0c76374-4140-42c2-9a0d-48a94a0e04b7\") " Oct 30 01:29:37.157702 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.157648 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/127ad507-8fcc-4c89-be18-27f08b405f8d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/127ad507-8fcc-4c89-be18-27f08b405f8d-maas-api-tls\") pod \"127ad507-8fcc-4c89-be18-27f08b405f8d\" (UID: \"127ad507-8fcc-4c89-be18-27f08b405f8d\") " Oct 30 01:29:37.157892 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.157730 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0c76374-4140-42c2-9a0d-48a94a0e04b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0c76374-4140-42c2-9a0d-48a94a0e04b7-plugins-config-volume\") pod \"c0c76374-4140-42c2-9a0d-48a94a0e04b7\" (UID: \"c0c76374-4140-42c2-9a0d-48a94a0e04b7\") " Oct 30 01:29:37.160175 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.160145 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/127ad507-8fcc-4c89-be18-27f08b405f8d-openshift-service-ca" pod "127ad507-8fcc-4c89-be18-27f08b405f8d" (UID: "127ad507-8fcc-4c89-be18-27f08b405f8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:37.163848 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.163524 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/127ad507-8fcc-4c89-be18-27f08b405f8d-kube-api-access-k77xh" pod "127ad507-8fcc-4c89-be18-27f08b405f8d" (UID: "127ad507-8fcc-4c89-be18-27f08b405f8d"). InnerVolumeSpecName "kube-api-access-k77xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:37.167026 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.166991 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c76374-4140-42c2-9a0d-48a94a0e04b7-plugins-config-volume" pod "c0c76374-4140-42c2-9a0d-48a94a0e04b7" (UID: "c0c76374-4140-42c2-9a0d-48a94a0e04b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:37.203045 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.201351 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/127ad507-8fcc-4c89-be18-27f08b405f8d-metrics-certs" pod "127ad507-8fcc-4c89-be18-27f08b405f8d" (UID: "127ad507-8fcc-4c89-be18-27f08b405f8d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:37.203045 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.201478 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c76374-4140-42c2-9a0d-48a94a0e04b7-kube-api-access-ls25m" pod "c0c76374-4140-42c2-9a0d-48a94a0e04b7" (UID: "c0c76374-4140-42c2-9a0d-48a94a0e04b7"). InnerVolumeSpecName "kube-api-access-ls25m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:37.210554 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.208935 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/127ad507-8fcc-4c89-be18-27f08b405f8d-maas-api-tls" pod "127ad507-8fcc-4c89-be18-27f08b405f8d" (UID: "127ad507-8fcc-4c89-be18-27f08b405f8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:37.259754 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.259713 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls25m\" (UniqueName: \"kubernetes.io/projected/c0c76374-4140-42c2-9a0d-48a94a0e04b7-kube-api-access-ls25m\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:37.259754 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.259749 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/127ad507-8fcc-4c89-be18-27f08b405f8d-maas-api-tls\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:37.259754 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.259774 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0c76374-4140-42c2-9a0d-48a94a0e04b7-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:37.260087 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.259785 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k77xh\" (UniqueName: \"kubernetes.io/projected/127ad507-8fcc-4c89-be18-27f08b405f8d-kube-api-access-k77xh\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:37.260087 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.259795 5300 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/127ad507-8fcc-4c89-be18-27f08b405f8d-metrics-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:37.260087 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.259807 5300 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/127ad507-8fcc-4c89-be18-27f08b405f8d-openshift-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:37.463588 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.463480 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:37.566938 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.566907 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:37.676252 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.676212 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:37.721091 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.721057 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw" Oct 30 01:29:37.721091 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.721072 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw" event={"ID":"c0c76374-4140-42c2-9a0d-48a94a0e04b7","Type":"ContainerDied","Data":"2994c4ae93e18edc6c778f6296ba22bf14aa875a9e087bc4b1b979300901b664"} Oct 30 01:29:37.721367 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.721134 5300 scope.go:122] "RemoveContainer" containerID="0b4e588f70888c3c55e3f54b6246afb20e4073590ae6c8f360eedf09dec0081b" Oct 30 01:29:37.723144 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.723117 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4" Oct 30 01:29:37.723276 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.723142 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4" event={"ID":"8af3eb07-ad1d-4484-a586-5a3f71386369","Type":"ContainerDied","Data":"b7bf8d837f6754b5003064e83ebe270f6efb939b4266f50b8ad70e4a918ebf62"} Oct 30 01:29:37.724735 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.724709 5300 generic.go:356] "Generic (PLEG): container finished" podID="127ad507-8fcc-4c89-be18-27f08b405f8d" containerID="579f38bd1fffebcda0bb22c366c83b2150c2675762942929d16786a9bfd17799" exitCode=0 Oct 30 01:29:37.724941 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.724823 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd" Oct 30 01:29:37.724941 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.724881 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd" event={"ID":"127ad507-8fcc-4c89-be18-27f08b405f8d","Type":"ContainerDied","Data":"579f38bd1fffebcda0bb22c366c83b2150c2675762942929d16786a9bfd17799"} Oct 30 01:29:37.724941 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.724920 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd" event={"ID":"127ad507-8fcc-4c89-be18-27f08b405f8d","Type":"ContainerDied","Data":"ce9fb5cc5877bf549f0e822571b00158c0cfbd1f5458dda865640b07bd90838e"} Oct 30 01:29:37.735616 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.735594 5300 scope.go:122] "RemoveContainer" containerID="dc0aad0bb30f7ca076b32b0d380d8eb1468cbb3a400d7da1b61790b3b7cecfdc" Oct 30 01:29:37.747659 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.747634 5300 scope.go:122] "RemoveContainer" containerID="579f38bd1fffebcda0bb22c366c83b2150c2675762942929d16786a9bfd17799" Oct 30 01:29:37.760129 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.760105 5300 scope.go:122] "RemoveContainer" containerID="579f38bd1fffebcda0bb22c366c83b2150c2675762942929d16786a9bfd17799" Oct 30 01:29:37.760465 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:29:37.760443 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"579f38bd1fffebcda0bb22c366c83b2150c2675762942929d16786a9bfd17799\": container with ID starting with 579f38bd1fffebcda0bb22c366c83b2150c2675762942929d16786a9bfd17799 not found: ID does not exist" containerID="579f38bd1fffebcda0bb22c366c83b2150c2675762942929d16786a9bfd17799" Oct 30 01:29:37.760564 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.760478 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"579f38bd1fffebcda0bb22c366c83b2150c2675762942929d16786a9bfd17799"} err="failed to get container status \"579f38bd1fffebcda0bb22c366c83b2150c2675762942929d16786a9bfd17799\": rpc error: code = NotFound desc = could not find container \"579f38bd1fffebcda0bb22c366c83b2150c2675762942929d16786a9bfd17799\": container with ID starting with 579f38bd1fffebcda0bb22c366c83b2150c2675762942929d16786a9bfd17799 not found: ID does not exist" Oct 30 01:29:37.826053 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.826016 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49bcda0b-9448-4e57-8328-220ee6208164" path="/var/lib/kubelet/pods/49bcda0b-9448-4e57-8328-220ee6208164/volumes" Oct 30 01:29:37.826373 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.826359 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw"] Oct 30 01:29:37.826414 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.826391 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-add1bb-699bfcfff5-dbnlw"] Oct 30 01:29:37.861330 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.861293 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd"] Oct 30 01:29:37.865008 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.864979 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-add1bb-7594f9594c-vtjkd"] Oct 30 01:29:37.900493 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.900456 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4"] Oct 30 01:29:37.907196 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:37.907168 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-add1bb-f9bccd668-s9cp4"] Oct 30 01:29:38.066442 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:38.066304 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:38.331678 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:38.331568 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:38.534087 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:38.534055 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:39.243373 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.243330 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerName="main" containerID="cri-o://ec43d364f1d122ffcfb274c18867ad0763d34f21317b7b0fded794b4e7189f36" gracePeriod=45 Oct 30 01:29:39.299435 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.299342 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": dial tcp 10.133.0.67:8000: connect: connection refused" Oct 30 01:29:39.517447 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.517413 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 01:29:39.591924 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.591883 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3369c82d-796c-4d63-a4cd-0ea21d50124b-kube-api-access-84wfw\" (UniqueName: \"kubernetes.io/projected/3369c82d-796c-4d63-a4cd-0ea21d50124b-kube-api-access-84wfw\") pod \"3369c82d-796c-4d63-a4cd-0ea21d50124b\" (UID: \"3369c82d-796c-4d63-a4cd-0ea21d50124b\") " Oct 30 01:29:39.592125 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.591937 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-tmp-dir\") pod \"3369c82d-796c-4d63-a4cd-0ea21d50124b\" (UID: \"3369c82d-796c-4d63-a4cd-0ea21d50124b\") " Oct 30 01:29:39.592125 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.591999 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-dshm\") pod \"3369c82d-796c-4d63-a4cd-0ea21d50124b\" (UID: \"3369c82d-796c-4d63-a4cd-0ea21d50124b\") " Oct 30 01:29:39.592125 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.592109 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-home\" (UniqueName: \"kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-home\") pod \"3369c82d-796c-4d63-a4cd-0ea21d50124b\" (UID: \"3369c82d-796c-4d63-a4cd-0ea21d50124b\") " Oct 30 01:29:39.592311 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.592190 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-model-cache\") pod \"3369c82d-796c-4d63-a4cd-0ea21d50124b\" (UID: \"3369c82d-796c-4d63-a4cd-0ea21d50124b\") " Oct 30 01:29:39.592311 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.592245 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3369c82d-796c-4d63-a4cd-0ea21d50124b-tls-certs\" (UniqueName: \"kubernetes.io/secret/3369c82d-796c-4d63-a4cd-0ea21d50124b-tls-certs\") pod \"3369c82d-796c-4d63-a4cd-0ea21d50124b\" (UID: \"3369c82d-796c-4d63-a4cd-0ea21d50124b\") " Oct 30 01:29:39.592967 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.592580 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-tmp-dir" pod "3369c82d-796c-4d63-a4cd-0ea21d50124b" (UID: "3369c82d-796c-4d63-a4cd-0ea21d50124b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:39.592967 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.592626 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-home" pod "3369c82d-796c-4d63-a4cd-0ea21d50124b" (UID: "3369c82d-796c-4d63-a4cd-0ea21d50124b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:39.592967 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.592735 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-model-cache" pod "3369c82d-796c-4d63-a4cd-0ea21d50124b" (UID: "3369c82d-796c-4d63-a4cd-0ea21d50124b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:39.592967 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.592879 5300 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-tmp-dir\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:39.592967 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.592900 5300 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-home\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:39.592967 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.592914 5300 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-model-cache\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:39.595711 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.595687 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-dshm" pod "3369c82d-796c-4d63-a4cd-0ea21d50124b" (UID: "3369c82d-796c-4d63-a4cd-0ea21d50124b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:39.595894 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.595717 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3369c82d-796c-4d63-a4cd-0ea21d50124b-tls-certs" pod "3369c82d-796c-4d63-a4cd-0ea21d50124b" (UID: "3369c82d-796c-4d63-a4cd-0ea21d50124b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:39.596126 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.596103 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3369c82d-796c-4d63-a4cd-0ea21d50124b-kube-api-access-84wfw" pod "3369c82d-796c-4d63-a4cd-0ea21d50124b" (UID: "3369c82d-796c-4d63-a4cd-0ea21d50124b"). InnerVolumeSpecName "kube-api-access-84wfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:39.660558 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.660521 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:39.694677 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.694645 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84wfw\" (UniqueName: \"kubernetes.io/projected/3369c82d-796c-4d63-a4cd-0ea21d50124b-kube-api-access-84wfw\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:39.694677 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.694673 5300 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3369c82d-796c-4d63-a4cd-0ea21d50124b-dshm\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:39.694677 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.694684 5300 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3369c82d-796c-4d63-a4cd-0ea21d50124b-tls-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:39.743056 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.743019 5300 generic.go:356] "Generic (PLEG): container finished" podID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerID="ec43d364f1d122ffcfb274c18867ad0763d34f21317b7b0fded794b4e7189f36" exitCode=0 Oct 30 01:29:39.743234 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.743110 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" Oct 30 01:29:39.743234 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.743106 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" event={"ID":"3369c82d-796c-4d63-a4cd-0ea21d50124b","Type":"ContainerDied","Data":"ec43d364f1d122ffcfb274c18867ad0763d34f21317b7b0fded794b4e7189f36"} Oct 30 01:29:39.743234 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.743197 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn" event={"ID":"3369c82d-796c-4d63-a4cd-0ea21d50124b","Type":"ContainerDied","Data":"09dc9a4dc98755474a9de6c17772a22602b8d7e3e869efaaf3c09ddf13f47643"} Oct 30 01:29:39.743234 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.743222 5300 scope.go:122] "RemoveContainer" containerID="ec43d364f1d122ffcfb274c18867ad0763d34f21317b7b0fded794b4e7189f36" Oct 30 01:29:39.777510 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.777479 5300 scope.go:122] "RemoveContainer" containerID="ec43d364f1d122ffcfb274c18867ad0763d34f21317b7b0fded794b4e7189f36" Oct 30 01:29:39.777935 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:29:39.777906 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec43d364f1d122ffcfb274c18867ad0763d34f21317b7b0fded794b4e7189f36\": container with ID starting with ec43d364f1d122ffcfb274c18867ad0763d34f21317b7b0fded794b4e7189f36 not found: ID does not exist" containerID="ec43d364f1d122ffcfb274c18867ad0763d34f21317b7b0fded794b4e7189f36" Oct 30 01:29:39.778054 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.777945 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec43d364f1d122ffcfb274c18867ad0763d34f21317b7b0fded794b4e7189f36"} err="failed to get container status \"ec43d364f1d122ffcfb274c18867ad0763d34f21317b7b0fded794b4e7189f36\": rpc error: code = NotFound desc = could not find container \"ec43d364f1d122ffcfb274c18867ad0763d34f21317b7b0fded794b4e7189f36\": container with ID starting with ec43d364f1d122ffcfb274c18867ad0763d34f21317b7b0fded794b4e7189f36 not found: ID does not exist" Oct 30 01:29:39.801895 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.801860 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn"] Oct 30 01:29:39.804512 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.804487 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-fd3281/distinct-2-w2-fd3281-kserve-fd94cb844-jl7vn"] Oct 30 01:29:39.822104 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.822071 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="127ad507-8fcc-4c89-be18-27f08b405f8d" path="/var/lib/kubelet/pods/127ad507-8fcc-4c89-be18-27f08b405f8d/volumes" Oct 30 01:29:39.822554 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.822540 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" path="/var/lib/kubelet/pods/3369c82d-796c-4d63-a4cd-0ea21d50124b/volumes" Oct 30 01:29:39.822952 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.822938 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8af3eb07-ad1d-4484-a586-5a3f71386369" path="/var/lib/kubelet/pods/8af3eb07-ad1d-4484-a586-5a3f71386369/volumes" Oct 30 01:29:39.823646 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:39.823631 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0c76374-4140-42c2-9a0d-48a94a0e04b7" path="/var/lib/kubelet/pods/c0c76374-4140-42c2-9a0d-48a94a0e04b7/volumes" Oct 30 01:29:40.156001 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:40.155911 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7"] Oct 30 01:29:40.290414 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:40.290371 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:40.412137 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:40.412014 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:41.231022 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:41.230992 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:42.871371 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:42.871331 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl" podUID="d3fb7228-7fdb-4db8-a878-d6f902cbc170" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:29:42.961109 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:42.961072 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:43.874938 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:43.874897 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw" podUID="1fd8605b-ff02-4868-a951-8fe172766e88" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:29:44.107710 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:44.107673 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:44.252966 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:44.252916 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:44.431093 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:44.431046 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:44.689044 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:44.688934 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:44.823584 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:44.823554 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:44.917995 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:44.917965 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:45.031808 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:45.031774 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:45.170120 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:45.165338 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7" podUID="83ca74f3-ea7d-46b6-b954-f681f39ba286" containerName="payload-processing" containerID="cri-o://89f9f8e5b2a4b26e049efc989b8682bf971ac3b6d804e576f57651de3a636a97" gracePeriod=25 Oct 30 01:29:45.488817 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:45.488783 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:46.806068 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:46.806016 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:49.184101 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:49.184070 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:49.409931 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:49.409887 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7" podUID="83ca74f3-ea7d-46b6-b954-f681f39ba286" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:29:50.039987 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:50.039944 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:50.472711 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:50.472675 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q"] Oct 30 01:29:50.486646 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:50.486612 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg"] Oct 30 01:29:50.590983 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:50.590949 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:50.860495 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:50.860404 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:51.593520 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:29:51.593483 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:29:52.486880 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:52.486850 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:52.869345 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:52.869252 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl" podUID="d3fb7228-7fdb-4db8-a878-d6f902cbc170" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:29:53.873887 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:53.873846 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw" podUID="1fd8605b-ff02-4868-a951-8fe172766e88" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:29:54.566690 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:54.566658 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:54.850279 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:54.850045 5300 generic.go:356] "Generic (PLEG): container finished" podID="d3fb7228-7fdb-4db8-a878-d6f902cbc170" containerID="5af672de892d6565734d950bcf7e7423cdd4122c455ba611690e459ab2e35941" exitCode=0 Oct 30 01:29:54.850279 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:54.850139 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl" event={"ID":"d3fb7228-7fdb-4db8-a878-d6f902cbc170","Type":"ContainerDied","Data":"5af672de892d6565734d950bcf7e7423cdd4122c455ba611690e459ab2e35941"} Oct 30 01:29:54.852987 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:54.852953 5300 generic.go:356] "Generic (PLEG): container finished" podID="1fd8605b-ff02-4868-a951-8fe172766e88" containerID="a0e810e2c3bfc7ea003835e85ce6cdcae08cf85bb01a4d72a4df1df7e5b1afc3" exitCode=0 Oct 30 01:29:54.853116 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:54.852984 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw" event={"ID":"1fd8605b-ff02-4868-a951-8fe172766e88","Type":"ContainerDied","Data":"a0e810e2c3bfc7ea003835e85ce6cdcae08cf85bb01a4d72a4df1df7e5b1afc3"} Oct 30 01:29:54.926433 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:54.926401 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl" Oct 30 01:29:54.957525 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:54.957499 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw" Oct 30 01:29:54.957839 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:54.957820 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3fb7228-7fdb-4db8-a878-d6f902cbc170-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3fb7228-7fdb-4db8-a878-d6f902cbc170-plugins-config-volume\") pod \"d3fb7228-7fdb-4db8-a878-d6f902cbc170\" (UID: \"d3fb7228-7fdb-4db8-a878-d6f902cbc170\") " Oct 30 01:29:54.957941 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:54.957927 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3fb7228-7fdb-4db8-a878-d6f902cbc170-kube-api-access-wgcp9\" (UniqueName: \"kubernetes.io/projected/d3fb7228-7fdb-4db8-a878-d6f902cbc170-kube-api-access-wgcp9\") pod \"d3fb7228-7fdb-4db8-a878-d6f902cbc170\" (UID: \"d3fb7228-7fdb-4db8-a878-d6f902cbc170\") " Oct 30 01:29:54.958173 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:54.958156 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3fb7228-7fdb-4db8-a878-d6f902cbc170-plugins-config-volume" pod "d3fb7228-7fdb-4db8-a878-d6f902cbc170" (UID: "d3fb7228-7fdb-4db8-a878-d6f902cbc170"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:54.958310 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:54.958290 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3fb7228-7fdb-4db8-a878-d6f902cbc170-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:54.959946 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:54.959927 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3fb7228-7fdb-4db8-a878-d6f902cbc170-kube-api-access-wgcp9" pod "d3fb7228-7fdb-4db8-a878-d6f902cbc170" (UID: "d3fb7228-7fdb-4db8-a878-d6f902cbc170"). InnerVolumeSpecName "kube-api-access-wgcp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:55.059318 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:55.059285 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fd8605b-ff02-4868-a951-8fe172766e88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fd8605b-ff02-4868-a951-8fe172766e88-plugins-config-volume\") pod \"1fd8605b-ff02-4868-a951-8fe172766e88\" (UID: \"1fd8605b-ff02-4868-a951-8fe172766e88\") " Oct 30 01:29:55.059530 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:55.059398 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fd8605b-ff02-4868-a951-8fe172766e88-kube-api-access-s5hxt\" (UniqueName: \"kubernetes.io/projected/1fd8605b-ff02-4868-a951-8fe172766e88-kube-api-access-s5hxt\") pod \"1fd8605b-ff02-4868-a951-8fe172766e88\" (UID: \"1fd8605b-ff02-4868-a951-8fe172766e88\") " Oct 30 01:29:55.059736 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:55.059708 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fd8605b-ff02-4868-a951-8fe172766e88-plugins-config-volume" pod "1fd8605b-ff02-4868-a951-8fe172766e88" (UID: "1fd8605b-ff02-4868-a951-8fe172766e88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:55.060255 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:55.060232 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fd8605b-ff02-4868-a951-8fe172766e88-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:55.060255 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:55.060254 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgcp9\" (UniqueName: \"kubernetes.io/projected/d3fb7228-7fdb-4db8-a878-d6f902cbc170-kube-api-access-wgcp9\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:55.062391 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:55.062366 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd8605b-ff02-4868-a951-8fe172766e88-kube-api-access-s5hxt" pod "1fd8605b-ff02-4868-a951-8fe172766e88" (UID: "1fd8605b-ff02-4868-a951-8fe172766e88"). InnerVolumeSpecName "kube-api-access-s5hxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:55.161834 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:55.161728 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5hxt\" (UniqueName: \"kubernetes.io/projected/1fd8605b-ff02-4868-a951-8fe172766e88-kube-api-access-s5hxt\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:29:55.792356 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:55.792321 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:55.864431 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:55.864326 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw" event={"ID":"1fd8605b-ff02-4868-a951-8fe172766e88","Type":"ContainerDied","Data":"68890c48fffe8b57826d3c21f42743f15ece255bd70b386b32306c428258303d"} Oct 30 01:29:55.864431 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:55.864390 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw" Oct 30 01:29:55.864431 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:55.864403 5300 scope.go:122] "RemoveContainer" containerID="a0e810e2c3bfc7ea003835e85ce6cdcae08cf85bb01a4d72a4df1df7e5b1afc3" Oct 30 01:29:55.867820 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:55.867733 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl" event={"ID":"d3fb7228-7fdb-4db8-a878-d6f902cbc170","Type":"ContainerDied","Data":"9a966786a238c6db3f8108513936cb369c2cca3862acb88c9f32e4d67d677440"} Oct 30 01:29:55.868004 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:55.867880 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl" Oct 30 01:29:55.878904 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:55.878879 5300 scope.go:122] "RemoveContainer" containerID="5af672de892d6565734d950bcf7e7423cdd4122c455ba611690e459ab2e35941" Oct 30 01:29:55.917447 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:55.917395 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:55.921724 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:55.921698 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl"] Oct 30 01:29:55.935972 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:55.935945 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-fd3281-864d489c66-rkxzl"] Oct 30 01:29:55.962663 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:55.962624 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw"] Oct 30 01:29:55.968465 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:55.968435 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-fd3281-596b5fc97f-hs5xw"] Oct 30 01:29:56.606369 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:56.606332 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:56.701260 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:56.701225 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:56.801427 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:56.801394 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:57.822323 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:57.822288 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fd8605b-ff02-4868-a951-8fe172766e88" path="/var/lib/kubelet/pods/1fd8605b-ff02-4868-a951-8fe172766e88/volumes" Oct 30 01:29:57.822743 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:57.822729 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3fb7228-7fdb-4db8-a878-d6f902cbc170" path="/var/lib/kubelet/pods/d3fb7228-7fdb-4db8-a878-d6f902cbc170/volumes" Oct 30 01:29:58.120524 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:58.120433 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:58.626294 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:58.626259 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:58.727951 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:58.727913 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:58.833736 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:58.833704 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:59.216081 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:59.216045 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:59.408856 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:29:59.408813 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7" podUID="83ca74f3-ea7d-46b6-b954-f681f39ba286" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:30:00.198442 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.198381 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc"] Oct 30 01:30:00.199926 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.199894 5300 cpu_manager.go:414] "containerMap: removing container" podUID="d3fb7228-7fdb-4db8-a878-d6f902cbc170" containerName="payload-processing" Oct 30 01:30:00.199926 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.199926 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3fb7228-7fdb-4db8-a878-d6f902cbc170" containerName="payload-processing" Oct 30 01:30:00.200140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.199943 5300 cpu_manager.go:414] "containerMap: removing container" podUID="d13883c2-ec67-4526-985d-f22c6673853f" containerName="istio-proxy" Oct 30 01:30:00.200140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.199951 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d13883c2-ec67-4526-985d-f22c6673853f" containerName="istio-proxy" Oct 30 01:30:00.200140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.199969 5300 cpu_manager.go:414] "containerMap: removing container" podUID="53efc96a-851b-49fd-afee-b158c3a849b6" containerName="payload-processing" Oct 30 01:30:00.200140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.199977 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53efc96a-851b-49fd-afee-b158c3a849b6" containerName="payload-processing" Oct 30 01:30:00.200140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.199992 5300 cpu_manager.go:414] "containerMap: removing container" podUID="1fd8605b-ff02-4868-a951-8fe172766e88" containerName="payload-pre-processing" Oct 30 01:30:00.200140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200000 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd8605b-ff02-4868-a951-8fe172766e88" containerName="payload-pre-processing" Oct 30 01:30:00.200140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200015 5300 cpu_manager.go:414] "containerMap: removing container" podUID="8af3eb07-ad1d-4484-a586-5a3f71386369" containerName="payload-processing" Oct 30 01:30:00.200140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200022 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8af3eb07-ad1d-4484-a586-5a3f71386369" containerName="payload-processing" Oct 30 01:30:00.200140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200034 5300 cpu_manager.go:414] "containerMap: removing container" podUID="b03e0096-ff6f-4e17-b213-72a40602bf18" containerName="istio-proxy" Oct 30 01:30:00.200140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200041 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b03e0096-ff6f-4e17-b213-72a40602bf18" containerName="istio-proxy" Oct 30 01:30:00.200140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200053 5300 cpu_manager.go:414] "containerMap: removing container" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerName="main" Oct 30 01:30:00.200140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200060 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerName="main" Oct 30 01:30:00.200140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200070 5300 cpu_manager.go:414] "containerMap: removing container" podUID="ce797d22-c33f-4853-8c58-e014cd26bf5b" containerName="istio-proxy" Oct 30 01:30:00.200140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200077 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce797d22-c33f-4853-8c58-e014cd26bf5b" containerName="istio-proxy" Oct 30 01:30:00.200140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200094 5300 cpu_manager.go:414] "containerMap: removing container" podUID="cfe18c62-795e-43b0-826c-35b3e91d9890" containerName="istio-proxy" Oct 30 01:30:00.200140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200101 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfe18c62-795e-43b0-826c-35b3e91d9890" containerName="istio-proxy" Oct 30 01:30:00.200140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200109 5300 cpu_manager.go:414] "containerMap: removing container" podUID="c8749ea6-d001-45fd-b7bd-956be5364f8c" containerName="istio-proxy" Oct 30 01:30:00.200140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200116 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8749ea6-d001-45fd-b7bd-956be5364f8c" containerName="istio-proxy" Oct 30 01:30:00.200140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200129 5300 cpu_manager.go:414] "containerMap: removing container" podUID="5f37290a-2d88-43bc-8af1-64bff456c157" containerName="maas-api" Oct 30 01:30:00.200140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200136 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f37290a-2d88-43bc-8af1-64bff456c157" containerName="maas-api" Oct 30 01:30:00.200140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200147 5300 cpu_manager.go:414] "containerMap: removing container" podUID="49bcda0b-9448-4e57-8328-220ee6208164" containerName="maas-api" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200154 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49bcda0b-9448-4e57-8328-220ee6208164" containerName="maas-api" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200167 5300 cpu_manager.go:414] "containerMap: removing container" podUID="653f0ddb-957e-4dae-85c9-b5822892df36" containerName="istio-proxy" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200174 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="653f0ddb-957e-4dae-85c9-b5822892df36" containerName="istio-proxy" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200189 5300 cpu_manager.go:414] "containerMap: removing container" podUID="ce797d22-c33f-4853-8c58-e014cd26bf5b" containerName="istio-proxy" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200197 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce797d22-c33f-4853-8c58-e014cd26bf5b" containerName="istio-proxy" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200212 5300 cpu_manager.go:414] "containerMap: removing container" podUID="127ad507-8fcc-4c89-be18-27f08b405f8d" containerName="maas-api" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200219 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="127ad507-8fcc-4c89-be18-27f08b405f8d" containerName="maas-api" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200231 5300 cpu_manager.go:414] "containerMap: removing container" podUID="c0c76374-4140-42c2-9a0d-48a94a0e04b7" containerName="payload-pre-processing" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200238 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0c76374-4140-42c2-9a0d-48a94a0e04b7" containerName="payload-pre-processing" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200256 5300 cpu_manager.go:414] "containerMap: removing container" podUID="26ccc4a1-e4a8-492b-abef-87dd46d9dab3" containerName="maas-api" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200260 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ccc4a1-e4a8-492b-abef-87dd46d9dab3" containerName="maas-api" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200274 5300 cpu_manager.go:414] "containerMap: removing container" podUID="151eb97e-bb5b-44ca-89c9-07025f5cb2d6" containerName="maas-api" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200278 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="151eb97e-bb5b-44ca-89c9-07025f5cb2d6" containerName="maas-api" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200286 5300 cpu_manager.go:414] "containerMap: removing container" podUID="ffd43904-bf14-4a9c-a752-d18c1fcff864" containerName="maas-api" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200290 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd43904-bf14-4a9c-a752-d18c1fcff864" containerName="maas-api" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200299 5300 cpu_manager.go:414] "containerMap: removing container" podUID="6b19732a-da65-4b29-a2e7-a2017ce9b815" containerName="payload-pre-processing" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200304 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b19732a-da65-4b29-a2e7-a2017ce9b815" containerName="payload-pre-processing" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200313 5300 cpu_manager.go:414] "containerMap: removing container" podUID="5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001" containerName="maas-api" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200317 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001" containerName="maas-api" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200322 5300 cpu_manager.go:414] "containerMap: removing container" podUID="26bb7124-a395-4690-a7dc-9d2236e9a083" containerName="payload-processing" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200327 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26bb7124-a395-4690-a7dc-9d2236e9a083" containerName="payload-processing" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200338 5300 cpu_manager.go:414] "containerMap: removing container" podUID="488a9458-8639-4ae8-9cd4-c839b1662099" containerName="istio-proxy" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200342 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="488a9458-8639-4ae8-9cd4-c839b1662099" containerName="istio-proxy" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200349 5300 cpu_manager.go:414] "containerMap: removing container" podUID="86e79fad-5a98-4840-9218-41ddff154453" containerName="istio-proxy" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200354 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86e79fad-5a98-4840-9218-41ddff154453" containerName="istio-proxy" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200367 5300 cpu_manager.go:414] "containerMap: removing container" podUID="aced2f61-18a5-4919-88a5-a76f6952085d" containerName="payload-pre-processing" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200374 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aced2f61-18a5-4919-88a5-a76f6952085d" containerName="payload-pre-processing" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200383 5300 cpu_manager.go:414] "containerMap: removing container" podUID="3f305e4e-92ce-4b37-adb8-3a8106e3854e" containerName="payload-processing" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200389 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f305e4e-92ce-4b37-adb8-3a8106e3854e" containerName="payload-processing" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200400 5300 cpu_manager.go:414] "containerMap: removing container" podUID="1455aae2-7d1f-4788-9a94-7469bee17334" containerName="istio-proxy" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200404 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1455aae2-7d1f-4788-9a94-7469bee17334" containerName="istio-proxy" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200514 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b19732a-da65-4b29-a2e7-a2017ce9b815" containerName="payload-pre-processing" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200527 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a0eb1e1-8d79-4ace-a2d1-f1f5383a3001" containerName="maas-api" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200533 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="26bb7124-a395-4690-a7dc-9d2236e9a083" containerName="payload-processing" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200543 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="488a9458-8639-4ae8-9cd4-c839b1662099" containerName="istio-proxy" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200550 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="86e79fad-5a98-4840-9218-41ddff154453" containerName="istio-proxy" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200556 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="aced2f61-18a5-4919-88a5-a76f6952085d" containerName="payload-pre-processing" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200563 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f305e4e-92ce-4b37-adb8-3a8106e3854e" containerName="payload-processing" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200569 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="1455aae2-7d1f-4788-9a94-7469bee17334" containerName="istio-proxy" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200575 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3fb7228-7fdb-4db8-a878-d6f902cbc170" containerName="payload-processing" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200583 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="d13883c2-ec67-4526-985d-f22c6673853f" containerName="istio-proxy" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200589 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="53efc96a-851b-49fd-afee-b158c3a849b6" containerName="payload-processing" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200594 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fd8605b-ff02-4868-a951-8fe172766e88" containerName="payload-pre-processing" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200601 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="8af3eb07-ad1d-4484-a586-5a3f71386369" containerName="payload-processing" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200606 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="b03e0096-ff6f-4e17-b213-72a40602bf18" containerName="istio-proxy" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200611 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="3369c82d-796c-4d63-a4cd-0ea21d50124b" containerName="main" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200617 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce797d22-c33f-4853-8c58-e014cd26bf5b" containerName="istio-proxy" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200622 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfe18c62-795e-43b0-826c-35b3e91d9890" containerName="istio-proxy" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200629 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8749ea6-d001-45fd-b7bd-956be5364f8c" containerName="istio-proxy" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200636 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f37290a-2d88-43bc-8af1-64bff456c157" containerName="maas-api" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200641 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="49bcda0b-9448-4e57-8328-220ee6208164" containerName="maas-api" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200646 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="653f0ddb-957e-4dae-85c9-b5822892df36" containerName="istio-proxy" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200652 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce797d22-c33f-4853-8c58-e014cd26bf5b" containerName="istio-proxy" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200659 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="127ad507-8fcc-4c89-be18-27f08b405f8d" containerName="maas-api" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200667 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0c76374-4140-42c2-9a0d-48a94a0e04b7" containerName="payload-pre-processing" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200674 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ccc4a1-e4a8-492b-abef-87dd46d9dab3" containerName="maas-api" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200679 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="d13883c2-ec67-4526-985d-f22c6673853f" containerName="istio-proxy" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200689 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="151eb97e-bb5b-44ca-89c9-07025f5cb2d6" containerName="maas-api" Oct 30 01:30:00.201257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.200696 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd43904-bf14-4a9c-a752-d18c1fcff864" containerName="maas-api" Oct 30 01:30:00.205985 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.205950 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92"] Oct 30 01:30:00.206107 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.206066 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc" Oct 30 01:30:00.206846 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.206824 5300 cpu_manager.go:414] "containerMap: removing container" podUID="d13883c2-ec67-4526-985d-f22c6673853f" containerName="istio-proxy" Oct 30 01:30:00.206846 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.206846 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d13883c2-ec67-4526-985d-f22c6673853f" containerName="istio-proxy" Oct 30 01:30:00.210483 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.210466 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92" Oct 30 01:30:00.233517 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.233484 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e15f0895-ce3b-401d-af86-68670b59de95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc\" (UID: \"e15f0895-ce3b-401d-af86-68670b59de95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc" Oct 30 01:30:00.233688 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.233539 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d4q8\" (UniqueName: \"kubernetes.io/projected/789a14f0-e4ff-4e2e-b587-3ed5cca21d54-kube-api-access-9d4q8\") pod \"maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92\" (UID: \"789a14f0-e4ff-4e2e-b587-3ed5cca21d54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92" Oct 30 01:30:00.233688 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.233573 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g9rz\" (UniqueName: \"kubernetes.io/projected/e15f0895-ce3b-401d-af86-68670b59de95-kube-api-access-4g9rz\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc\" (UID: \"e15f0895-ce3b-401d-af86-68670b59de95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc" Oct 30 01:30:00.233849 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.233799 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/789a14f0-e4ff-4e2e-b587-3ed5cca21d54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92\" (UID: \"789a14f0-e4ff-4e2e-b587-3ed5cca21d54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92" Oct 30 01:30:00.238306 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.238279 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc"] Oct 30 01:30:00.246697 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.246621 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92"] Oct 30 01:30:00.251568 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.251539 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-mmvfp"] Oct 30 01:30:00.260211 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.260188 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv"] Oct 30 01:30:00.262124 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.261856 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-mmvfp" Oct 30 01:30:00.268609 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.268582 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv"] Oct 30 01:30:00.268835 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.268697 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv" Oct 30 01:30:00.283355 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.283332 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqvx9\"" Oct 30 01:30:00.286739 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.286703 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-mmvfp"] Oct 30 01:30:00.334586 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.334548 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn5wh\" (UniqueName: \"kubernetes.io/projected/2c39939a-6ce7-4527-b066-ce41e01d74d4-kube-api-access-fn5wh\") pod \"maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv\" (UID: \"2c39939a-6ce7-4527-b066-ce41e01d74d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv" Oct 30 01:30:00.334822 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.334699 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888730-mmvfp\" (UID: \"c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-mmvfp" Oct 30 01:30:00.334822 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.334741 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e15f0895-ce3b-401d-af86-68670b59de95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc\" (UID: \"e15f0895-ce3b-401d-af86-68670b59de95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc" Oct 30 01:30:00.334953 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.334854 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d4q8\" (UniqueName: \"kubernetes.io/projected/789a14f0-e4ff-4e2e-b587-3ed5cca21d54-kube-api-access-9d4q8\") pod \"maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92\" (UID: \"789a14f0-e4ff-4e2e-b587-3ed5cca21d54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92" Oct 30 01:30:00.334953 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.334907 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g9rz\" (UniqueName: \"kubernetes.io/projected/e15f0895-ce3b-401d-af86-68670b59de95-kube-api-access-4g9rz\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc\" (UID: \"e15f0895-ce3b-401d-af86-68670b59de95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc" Oct 30 01:30:00.335140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.335116 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/789a14f0-e4ff-4e2e-b587-3ed5cca21d54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92\" (UID: \"789a14f0-e4ff-4e2e-b587-3ed5cca21d54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92" Oct 30 01:30:00.335279 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.335181 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c39939a-6ce7-4527-b066-ce41e01d74d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv\" (UID: \"2c39939a-6ce7-4527-b066-ce41e01d74d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv" Oct 30 01:30:00.335279 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.335226 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl2n8\" (UniqueName: \"kubernetes.io/projected/c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a-kube-api-access-bl2n8\") pod \"maas-api-key-cleanup-29888730-mmvfp\" (UID: \"c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-mmvfp" Oct 30 01:30:00.335813 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.335741 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e15f0895-ce3b-401d-af86-68670b59de95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc\" (UID: \"e15f0895-ce3b-401d-af86-68670b59de95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc" Oct 30 01:30:00.336104 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.336080 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/789a14f0-e4ff-4e2e-b587-3ed5cca21d54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92\" (UID: \"789a14f0-e4ff-4e2e-b587-3ed5cca21d54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92" Oct 30 01:30:00.368672 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.368640 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g9rz\" (UniqueName: \"kubernetes.io/projected/e15f0895-ce3b-401d-af86-68670b59de95-kube-api-access-4g9rz\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc\" (UID: \"e15f0895-ce3b-401d-af86-68670b59de95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc" Oct 30 01:30:00.368910 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.368688 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d4q8\" (UniqueName: \"kubernetes.io/projected/789a14f0-e4ff-4e2e-b587-3ed5cca21d54-kube-api-access-9d4q8\") pod \"maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92\" (UID: \"789a14f0-e4ff-4e2e-b587-3ed5cca21d54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92" Oct 30 01:30:00.436389 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.436356 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn5wh\" (UniqueName: \"kubernetes.io/projected/2c39939a-6ce7-4527-b066-ce41e01d74d4-kube-api-access-fn5wh\") pod \"maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv\" (UID: \"2c39939a-6ce7-4527-b066-ce41e01d74d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv" Oct 30 01:30:00.436599 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.436436 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888730-mmvfp\" (UID: \"c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-mmvfp" Oct 30 01:30:00.436599 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.436581 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c39939a-6ce7-4527-b066-ce41e01d74d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv\" (UID: \"2c39939a-6ce7-4527-b066-ce41e01d74d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv" Oct 30 01:30:00.436743 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.436606 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl2n8\" (UniqueName: \"kubernetes.io/projected/c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a-kube-api-access-bl2n8\") pod \"maas-api-key-cleanup-29888730-mmvfp\" (UID: \"c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-mmvfp" Oct 30 01:30:00.437200 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.437174 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888730-mmvfp\" (UID: \"c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-mmvfp" Oct 30 01:30:00.437288 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.437196 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c39939a-6ce7-4527-b066-ce41e01d74d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv\" (UID: \"2c39939a-6ce7-4527-b066-ce41e01d74d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv" Oct 30 01:30:00.459107 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.459032 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl2n8\" (UniqueName: \"kubernetes.io/projected/c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a-kube-api-access-bl2n8\") pod \"maas-api-key-cleanup-29888730-mmvfp\" (UID: \"c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-mmvfp" Oct 30 01:30:00.459783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.459744 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn5wh\" (UniqueName: \"kubernetes.io/projected/2c39939a-6ce7-4527-b066-ce41e01d74d4-kube-api-access-fn5wh\") pod \"maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv\" (UID: \"2c39939a-6ce7-4527-b066-ce41e01d74d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv" Oct 30 01:30:00.532443 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.532409 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc" Oct 30 01:30:00.548989 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.548960 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92" Oct 30 01:30:00.554962 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.554922 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:00.585813 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.585779 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-mmvfp" Oct 30 01:30:00.596937 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.596910 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv" Oct 30 01:30:00.856826 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.856053 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc"] Oct 30 01:30:00.895826 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.895354 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92"] Oct 30 01:30:00.905999 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.905972 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc" event={"ID":"e15f0895-ce3b-401d-af86-68670b59de95","Type":"ContainerStarted","Data":"da6f5d0b858631bd4133e4be61496529f0563ee2fa44dc6f1516e0a27f15a6da"} Oct 30 01:30:00.953955 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:00.953925 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-mmvfp"] Oct 30 01:30:00.955840 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:30:00.955806 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc097aa62_d0cc_4f5d_9d98_c5a9133f4d3a.slice/crio-509753fbfa589748d953b39578bc7ad627c417dccb92e3e46f723fef89004ef0 WatchSource:0}: Error finding container 509753fbfa589748d953b39578bc7ad627c417dccb92e3e46f723fef89004ef0: Status 404 returned error can't find the container with id 509753fbfa589748d953b39578bc7ad627c417dccb92e3e46f723fef89004ef0 Oct 30 01:30:01.004134 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:01.004106 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv"] Oct 30 01:30:01.005001 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:30:01.004979 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c39939a_6ce7_4527_b066_ce41e01d74d4.slice/crio-7b4dc6d8b1101dab352738970b91dca9a34a4c9f345656998e3a7efd28949e4f WatchSource:0}: Error finding container 7b4dc6d8b1101dab352738970b91dca9a34a4c9f345656998e3a7efd28949e4f: Status 404 returned error can't find the container with id 7b4dc6d8b1101dab352738970b91dca9a34a4c9f345656998e3a7efd28949e4f Oct 30 01:30:01.327165 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:01.327121 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:01.426526 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:01.426490 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:01.914915 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:01.914883 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv" event={"ID":"2c39939a-6ce7-4527-b066-ce41e01d74d4","Type":"ContainerStarted","Data":"a99a193e42c9dc932f488c2fe11c2da370f5ef6930653bad37d22d73e0fb296d"} Oct 30 01:30:01.915070 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:01.914942 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv" event={"ID":"2c39939a-6ce7-4527-b066-ce41e01d74d4","Type":"ContainerStarted","Data":"7b4dc6d8b1101dab352738970b91dca9a34a4c9f345656998e3a7efd28949e4f"} Oct 30 01:30:01.916733 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:01.916706 5300 generic.go:356] "Generic (PLEG): container finished" podID="c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a" containerID="52ee3c5572c08666098abfa8efae59cc85e2deb118cb3d01149ba7000463a23f" exitCode=60 Oct 30 01:30:01.916874 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:01.916815 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-mmvfp" event={"ID":"c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a","Type":"ContainerDied","Data":"52ee3c5572c08666098abfa8efae59cc85e2deb118cb3d01149ba7000463a23f"} Oct 30 01:30:01.916874 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:01.916855 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-mmvfp" event={"ID":"c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a","Type":"ContainerStarted","Data":"509753fbfa589748d953b39578bc7ad627c417dccb92e3e46f723fef89004ef0"} Oct 30 01:30:01.917080 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:01.917061 5300 scope.go:122] "RemoveContainer" containerID="52ee3c5572c08666098abfa8efae59cc85e2deb118cb3d01149ba7000463a23f" Oct 30 01:30:01.921741 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:01.921695 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92" event={"ID":"789a14f0-e4ff-4e2e-b587-3ed5cca21d54","Type":"ContainerStarted","Data":"f249b0454678a449104c21797885b6f75b3eed21b9afa0a8c4093609de28d090"} Oct 30 01:30:01.921990 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:01.921744 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92" event={"ID":"789a14f0-e4ff-4e2e-b587-3ed5cca21d54","Type":"ContainerStarted","Data":"5463f57e5a6675a1b0fd82196fcf49c908ae28493dfe9e3bd2c2f43d6768814a"} Oct 30 01:30:01.926007 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:01.924866 5300 generic.go:356] "Generic (PLEG): container finished" podID="e15f0895-ce3b-401d-af86-68670b59de95" containerID="89457512b1ffada4610bb797b11388ae32587572cc0a8f3bccdec587cd2b7a4e" exitCode=60 Oct 30 01:30:01.926007 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:01.925062 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc" event={"ID":"e15f0895-ce3b-401d-af86-68670b59de95","Type":"ContainerDied","Data":"89457512b1ffada4610bb797b11388ae32587572cc0a8f3bccdec587cd2b7a4e"} Oct 30 01:30:01.926007 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:01.925564 5300 scope.go:122] "RemoveContainer" containerID="89457512b1ffada4610bb797b11388ae32587572cc0a8f3bccdec587cd2b7a4e" Oct 30 01:30:02.007530 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:02.007488 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv" podStartSLOduration=2.007464253 podStartE2EDuration="2.007464253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:01.963106356 +0000 UTC m=+2802.278974683" watchObservedRunningTime="2026-10-30 01:30:02.007464253 +0000 UTC m=+2802.323332578" Oct 30 01:30:02.060541 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:02.060495 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92" podStartSLOduration=2.060479061 podStartE2EDuration="2.060479061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:02.059409247 +0000 UTC m=+2802.375277571" watchObservedRunningTime="2026-10-30 01:30:02.060479061 +0000 UTC m=+2802.376347385" Oct 30 01:30:02.473697 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:02.473661 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:02.934107 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:02.934077 5300 generic.go:356] "Generic (PLEG): container finished" podID="2c39939a-6ce7-4527-b066-ce41e01d74d4" containerID="a99a193e42c9dc932f488c2fe11c2da370f5ef6930653bad37d22d73e0fb296d" exitCode=60 Oct 30 01:30:02.934232 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:02.934160 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv" event={"ID":"2c39939a-6ce7-4527-b066-ce41e01d74d4","Type":"ContainerDied","Data":"a99a193e42c9dc932f488c2fe11c2da370f5ef6930653bad37d22d73e0fb296d"} Oct 30 01:30:02.934485 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:02.934470 5300 scope.go:122] "RemoveContainer" containerID="a99a193e42c9dc932f488c2fe11c2da370f5ef6930653bad37d22d73e0fb296d" Oct 30 01:30:02.936218 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:02.936198 5300 generic.go:356] "Generic (PLEG): container finished" podID="c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a" containerID="da168eaca863c9deb7e5b3bc7d821fc5ddf5f58430684a6949be87fc81103b8b" exitCode=60 Oct 30 01:30:02.936285 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:02.936273 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-mmvfp" event={"ID":"c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a","Type":"ContainerDied","Data":"da168eaca863c9deb7e5b3bc7d821fc5ddf5f58430684a6949be87fc81103b8b"} Oct 30 01:30:02.936348 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:02.936311 5300 scope.go:122] "RemoveContainer" containerID="52ee3c5572c08666098abfa8efae59cc85e2deb118cb3d01149ba7000463a23f" Oct 30 01:30:02.936632 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:02.936614 5300 scope.go:122] "RemoveContainer" containerID="da168eaca863c9deb7e5b3bc7d821fc5ddf5f58430684a6949be87fc81103b8b" Oct 30 01:30:02.936973 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:30:02.936947 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888730-mmvfp_odh-ai-gateway-infra(c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-mmvfp" podUID="c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a" Oct 30 01:30:02.938119 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:02.938096 5300 generic.go:356] "Generic (PLEG): container finished" podID="789a14f0-e4ff-4e2e-b587-3ed5cca21d54" containerID="f249b0454678a449104c21797885b6f75b3eed21b9afa0a8c4093609de28d090" exitCode=60 Oct 30 01:30:02.938205 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:02.938172 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92" event={"ID":"789a14f0-e4ff-4e2e-b587-3ed5cca21d54","Type":"ContainerDied","Data":"f249b0454678a449104c21797885b6f75b3eed21b9afa0a8c4093609de28d090"} Oct 30 01:30:02.938417 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:02.938402 5300 scope.go:122] "RemoveContainer" containerID="f249b0454678a449104c21797885b6f75b3eed21b9afa0a8c4093609de28d090" Oct 30 01:30:02.940035 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:02.940016 5300 generic.go:356] "Generic (PLEG): container finished" podID="e15f0895-ce3b-401d-af86-68670b59de95" containerID="19043f6c80ca637f59e2078af8b185572aa410ac7a26fe8f74977b9b2cfe8a83" exitCode=60 Oct 30 01:30:02.940145 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:02.940096 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc" event={"ID":"e15f0895-ce3b-401d-af86-68670b59de95","Type":"ContainerDied","Data":"19043f6c80ca637f59e2078af8b185572aa410ac7a26fe8f74977b9b2cfe8a83"} Oct 30 01:30:02.940380 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:02.940365 5300 scope.go:122] "RemoveContainer" containerID="19043f6c80ca637f59e2078af8b185572aa410ac7a26fe8f74977b9b2cfe8a83" Oct 30 01:30:02.940588 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:30:02.940570 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc_odh-ai-gateway-infra(e15f0895-ce3b-401d-af86-68670b59de95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc" podUID="e15f0895-ce3b-401d-af86-68670b59de95" Oct 30 01:30:03.051128 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:03.051107 5300 scope.go:122] "RemoveContainer" containerID="89457512b1ffada4610bb797b11388ae32587572cc0a8f3bccdec587cd2b7a4e" Oct 30 01:30:03.183377 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:03.183295 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:03.767471 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:03.767412 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:03.944549 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:03.944500 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:03.952735 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:03.952697 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv" event={"ID":"2c39939a-6ce7-4527-b066-ce41e01d74d4","Type":"ContainerStarted","Data":"fca08f3cef1ddb9b13f8e82c595abd33410dad0b2eb32507a5088a06c5658848"} Oct 30 01:30:03.956172 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:03.956146 5300 scope.go:122] "RemoveContainer" containerID="da168eaca863c9deb7e5b3bc7d821fc5ddf5f58430684a6949be87fc81103b8b" Oct 30 01:30:03.956544 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:30:03.956519 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888730-mmvfp_odh-ai-gateway-infra(c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-mmvfp" podUID="c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a" Oct 30 01:30:03.957982 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:03.957961 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92" event={"ID":"789a14f0-e4ff-4e2e-b587-3ed5cca21d54","Type":"ContainerStarted","Data":"179085c3ea0d825f4c6a0d259b9f8d1d89580aeea79301d4bbecc03188ac6f96"} Oct 30 01:30:03.959611 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:03.959592 5300 scope.go:122] "RemoveContainer" containerID="19043f6c80ca637f59e2078af8b185572aa410ac7a26fe8f74977b9b2cfe8a83" Oct 30 01:30:03.959853 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:30:03.959833 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc_odh-ai-gateway-infra(e15f0895-ce3b-401d-af86-68670b59de95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc" podUID="e15f0895-ce3b-401d-af86-68670b59de95" Oct 30 01:30:04.969522 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:04.969483 5300 generic.go:356] "Generic (PLEG): container finished" podID="2c39939a-6ce7-4527-b066-ce41e01d74d4" containerID="fca08f3cef1ddb9b13f8e82c595abd33410dad0b2eb32507a5088a06c5658848" exitCode=60 Oct 30 01:30:04.970134 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:04.969581 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv" event={"ID":"2c39939a-6ce7-4527-b066-ce41e01d74d4","Type":"ContainerDied","Data":"fca08f3cef1ddb9b13f8e82c595abd33410dad0b2eb32507a5088a06c5658848"} Oct 30 01:30:04.970134 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:04.969651 5300 scope.go:122] "RemoveContainer" containerID="a99a193e42c9dc932f488c2fe11c2da370f5ef6930653bad37d22d73e0fb296d" Oct 30 01:30:04.970134 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:04.969929 5300 scope.go:122] "RemoveContainer" containerID="fca08f3cef1ddb9b13f8e82c595abd33410dad0b2eb32507a5088a06c5658848" Oct 30 01:30:04.970416 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:30:04.970288 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv_odh-ai-gateway-infra(2c39939a-6ce7-4527-b066-ce41e01d74d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv" podUID="2c39939a-6ce7-4527-b066-ce41e01d74d4" Oct 30 01:30:04.971932 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:04.971912 5300 generic.go:356] "Generic (PLEG): container finished" podID="789a14f0-e4ff-4e2e-b587-3ed5cca21d54" containerID="179085c3ea0d825f4c6a0d259b9f8d1d89580aeea79301d4bbecc03188ac6f96" exitCode=60 Oct 30 01:30:04.972035 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:04.971994 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92" event={"ID":"789a14f0-e4ff-4e2e-b587-3ed5cca21d54","Type":"ContainerDied","Data":"179085c3ea0d825f4c6a0d259b9f8d1d89580aeea79301d4bbecc03188ac6f96"} Oct 30 01:30:04.972321 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:04.972307 5300 scope.go:122] "RemoveContainer" containerID="179085c3ea0d825f4c6a0d259b9f8d1d89580aeea79301d4bbecc03188ac6f96" Oct 30 01:30:04.972635 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:30:04.972618 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92_odh-ai-gateway-infra(789a14f0-e4ff-4e2e-b587-3ed5cca21d54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92" podUID="789a14f0-e4ff-4e2e-b587-3ed5cca21d54" Oct 30 01:30:04.982402 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:04.982382 5300 scope.go:122] "RemoveContainer" containerID="f249b0454678a449104c21797885b6f75b3eed21b9afa0a8c4093609de28d090" Oct 30 01:30:04.990310 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:04.990287 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:05.483938 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.483910 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7" Oct 30 01:30:05.496746 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.496712 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" podUID="d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" containerName="main" containerID="cri-o://18e1c9907ac4c4d1dcb3a430d9a8e869da48df7cd6688e69849d384f835eaa09" gracePeriod=45 Oct 30 01:30:05.510304 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.510247 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" containerID="cri-o://08a8a17422c6da0dda5722c19a464b5c5e4db161e9638a70759c8289e29709e0" gracePeriod=45 Oct 30 01:30:05.550187 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.550102 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83ca74f3-ea7d-46b6-b954-f681f39ba286-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83ca74f3-ea7d-46b6-b954-f681f39ba286-plugins-config-volume\") pod \"83ca74f3-ea7d-46b6-b954-f681f39ba286\" (UID: \"83ca74f3-ea7d-46b6-b954-f681f39ba286\") " Oct 30 01:30:05.550187 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.550174 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83ca74f3-ea7d-46b6-b954-f681f39ba286-kube-api-access-4gr68\" (UniqueName: \"kubernetes.io/projected/83ca74f3-ea7d-46b6-b954-f681f39ba286-kube-api-access-4gr68\") pod \"83ca74f3-ea7d-46b6-b954-f681f39ba286\" (UID: \"83ca74f3-ea7d-46b6-b954-f681f39ba286\") " Oct 30 01:30:05.550580 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.550550 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83ca74f3-ea7d-46b6-b954-f681f39ba286-plugins-config-volume" pod "83ca74f3-ea7d-46b6-b954-f681f39ba286" (UID: "83ca74f3-ea7d-46b6-b954-f681f39ba286"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:05.553670 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.551009 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83ca74f3-ea7d-46b6-b954-f681f39ba286-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.553670 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.553038 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ca74f3-ea7d-46b6-b954-f681f39ba286-kube-api-access-4gr68" pod "83ca74f3-ea7d-46b6-b954-f681f39ba286" (UID: "83ca74f3-ea7d-46b6-b954-f681f39ba286"). InnerVolumeSpecName "kube-api-access-4gr68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:05.652622 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.652588 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gr68\" (UniqueName: \"kubernetes.io/projected/83ca74f3-ea7d-46b6-b954-f681f39ba286-kube-api-access-4gr68\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.763288 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.763258 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 01:30:05.794153 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.794121 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 01:30:05.854741 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.854708 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-kube-api-access-hgj5b\" (UniqueName: \"kubernetes.io/projected/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-kube-api-access-hgj5b\") pod \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\" (UID: \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\") " Oct 30 01:30:05.854943 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.854774 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-tls-certs\" (UniqueName: \"kubernetes.io/secret/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-tls-certs\") pod \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\" (UID: \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\") " Oct 30 01:30:05.854943 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.854808 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-model-cache\") pod \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\" (UID: \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\") " Oct 30 01:30:05.854943 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.854896 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-dshm\") pod \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\" (UID: \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\") " Oct 30 01:30:05.855137 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.854989 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-model-cache\") pod \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\" (UID: \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\") " Oct 30 01:30:05.855137 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.855081 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-tmp-dir\") pod \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\" (UID: \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\") " Oct 30 01:30:05.855137 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.855130 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-tmp-dir\") pod \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\" (UID: \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\") " Oct 30 01:30:05.855316 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.855269 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-home\" (UniqueName: \"kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-home\") pod \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\" (UID: \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\") " Oct 30 01:30:05.855316 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.855298 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-kube-api-access-lv5bg\" (UniqueName: \"kubernetes.io/projected/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-kube-api-access-lv5bg\") pod \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\" (UID: \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\") " Oct 30 01:30:05.855423 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.855331 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-tls-certs\" (UniqueName: \"kubernetes.io/secret/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-tls-certs\") pod \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\" (UID: \"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b\") " Oct 30 01:30:05.855423 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.855359 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-home\" (UniqueName: \"kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-home\") pod \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\" (UID: \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\") " Oct 30 01:30:05.855423 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.855394 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-dshm\") pod \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\" (UID: \"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9\") " Oct 30 01:30:05.855596 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.855487 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-model-cache" pod "a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" (UID: "a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:05.856926 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.855841 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-tmp-dir" pod "d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" (UID: "d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:05.856926 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.856057 5300 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-tmp-dir\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.856926 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.856078 5300 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-model-cache\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.856926 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.856145 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-tmp-dir" pod "a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" (UID: "a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:05.856926 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.856425 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-model-cache" pod "d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" (UID: "d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:05.856926 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.856669 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-home" pod "a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" (UID: "a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:05.856926 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.856878 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-home" pod "d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" (UID: "d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:05.857947 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.857909 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-kube-api-access-hgj5b" pod "a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" (UID: "a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9"). InnerVolumeSpecName "kube-api-access-hgj5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:05.859200 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.859165 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-dshm" pod "a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" (UID: "a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:05.860721 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.860478 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-dshm" pod "d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" (UID: "d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:05.860721 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.860551 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-tls-certs" pod "a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" (UID: "a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:05.860721 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.860659 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-tls-certs" pod "d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" (UID: "d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:05.862590 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.862562 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-kube-api-access-lv5bg" pod "d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" (UID: "d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b"). InnerVolumeSpecName "kube-api-access-lv5bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:05.957612 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.957575 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgj5b\" (UniqueName: \"kubernetes.io/projected/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-kube-api-access-hgj5b\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.957824 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.957687 5300 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-tls-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.957824 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.957712 5300 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-dshm\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.957824 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.957727 5300 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-model-cache\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.957824 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.957741 5300 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-tmp-dir\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.957824 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.957750 5300 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-home\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.957824 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.957772 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv5bg\" (UniqueName: \"kubernetes.io/projected/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-kube-api-access-lv5bg\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.957824 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.957795 5300 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b-tls-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.957824 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.957810 5300 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-home\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.957824 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.957818 5300 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9-dshm\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:05.993191 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.993150 5300 generic.go:356] "Generic (PLEG): container finished" podID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerID="08a8a17422c6da0dda5722c19a464b5c5e4db161e9638a70759c8289e29709e0" exitCode=0 Oct 30 01:30:05.994001 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.993347 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" event={"ID":"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9","Type":"ContainerDied","Data":"08a8a17422c6da0dda5722c19a464b5c5e4db161e9638a70759c8289e29709e0"} Oct 30 01:30:05.994001 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.993389 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" event={"ID":"a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9","Type":"ContainerDied","Data":"f0816c29b647986ef4c9f38c0402bc2bb9d3ec5c7e5b1553f068922bc290b8ab"} Oct 30 01:30:05.994001 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.993414 5300 scope.go:122] "RemoveContainer" containerID="08a8a17422c6da0dda5722c19a464b5c5e4db161e9638a70759c8289e29709e0" Oct 30 01:30:05.994001 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.993434 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg" Oct 30 01:30:05.998610 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:05.998566 5300 scope.go:122] "RemoveContainer" containerID="fca08f3cef1ddb9b13f8e82c595abd33410dad0b2eb32507a5088a06c5658848" Oct 30 01:30:05.999179 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:30:05.998972 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv_odh-ai-gateway-infra(2c39939a-6ce7-4527-b066-ce41e01d74d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv" podUID="2c39939a-6ce7-4527-b066-ce41e01d74d4" Oct 30 01:30:06.006392 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.006361 5300 scope.go:122] "RemoveContainer" containerID="179085c3ea0d825f4c6a0d259b9f8d1d89580aeea79301d4bbecc03188ac6f96" Oct 30 01:30:06.006845 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:30:06.006813 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92_odh-ai-gateway-infra(789a14f0-e4ff-4e2e-b587-3ed5cca21d54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92" podUID="789a14f0-e4ff-4e2e-b587-3ed5cca21d54" Oct 30 01:30:06.009279 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.009252 5300 generic.go:356] "Generic (PLEG): container finished" podID="d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" containerID="18e1c9907ac4c4d1dcb3a430d9a8e869da48df7cd6688e69849d384f835eaa09" exitCode=0 Oct 30 01:30:06.009387 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.009295 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" Oct 30 01:30:06.009387 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.009348 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" event={"ID":"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b","Type":"ContainerDied","Data":"18e1c9907ac4c4d1dcb3a430d9a8e869da48df7cd6688e69849d384f835eaa09"} Oct 30 01:30:06.009387 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.009375 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q" event={"ID":"d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b","Type":"ContainerDied","Data":"6b7e4a393e22c460ddda9f8b7be32567910883c6a15aeff581fb6d6d6cdf335c"} Oct 30 01:30:06.010844 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.010821 5300 generic.go:356] "Generic (PLEG): container finished" podID="83ca74f3-ea7d-46b6-b954-f681f39ba286" containerID="89f9f8e5b2a4b26e049efc989b8682bf971ac3b6d804e576f57651de3a636a97" exitCode=0 Oct 30 01:30:06.010957 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.010865 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7" event={"ID":"83ca74f3-ea7d-46b6-b954-f681f39ba286","Type":"ContainerDied","Data":"89f9f8e5b2a4b26e049efc989b8682bf971ac3b6d804e576f57651de3a636a97"} Oct 30 01:30:06.010957 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.010896 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7" event={"ID":"83ca74f3-ea7d-46b6-b954-f681f39ba286","Type":"ContainerDied","Data":"07e58a422e6fd6df3f5ea8f51156e3a99747c35dfc33788d2d055aec357412a2"} Oct 30 01:30:06.010957 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.010939 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7" Oct 30 01:30:06.017906 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.017835 5300 scope.go:122] "RemoveContainer" containerID="08a8a17422c6da0dda5722c19a464b5c5e4db161e9638a70759c8289e29709e0" Oct 30 01:30:06.018693 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:30:06.018347 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08a8a17422c6da0dda5722c19a464b5c5e4db161e9638a70759c8289e29709e0\": container with ID starting with 08a8a17422c6da0dda5722c19a464b5c5e4db161e9638a70759c8289e29709e0 not found: ID does not exist" containerID="08a8a17422c6da0dda5722c19a464b5c5e4db161e9638a70759c8289e29709e0" Oct 30 01:30:06.018693 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.018381 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08a8a17422c6da0dda5722c19a464b5c5e4db161e9638a70759c8289e29709e0"} err="failed to get container status \"08a8a17422c6da0dda5722c19a464b5c5e4db161e9638a70759c8289e29709e0\": rpc error: code = NotFound desc = could not find container \"08a8a17422c6da0dda5722c19a464b5c5e4db161e9638a70759c8289e29709e0\": container with ID starting with 08a8a17422c6da0dda5722c19a464b5c5e4db161e9638a70759c8289e29709e0 not found: ID does not exist" Oct 30 01:30:06.018693 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.018401 5300 scope.go:122] "RemoveContainer" containerID="18e1c9907ac4c4d1dcb3a430d9a8e869da48df7cd6688e69849d384f835eaa09" Oct 30 01:30:06.031072 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.031040 5300 scope.go:122] "RemoveContainer" containerID="18e1c9907ac4c4d1dcb3a430d9a8e869da48df7cd6688e69849d384f835eaa09" Oct 30 01:30:06.031346 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:30:06.031326 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18e1c9907ac4c4d1dcb3a430d9a8e869da48df7cd6688e69849d384f835eaa09\": container with ID starting with 18e1c9907ac4c4d1dcb3a430d9a8e869da48df7cd6688e69849d384f835eaa09 not found: ID does not exist" containerID="18e1c9907ac4c4d1dcb3a430d9a8e869da48df7cd6688e69849d384f835eaa09" Oct 30 01:30:06.031427 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.031356 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18e1c9907ac4c4d1dcb3a430d9a8e869da48df7cd6688e69849d384f835eaa09"} err="failed to get container status \"18e1c9907ac4c4d1dcb3a430d9a8e869da48df7cd6688e69849d384f835eaa09\": rpc error: code = NotFound desc = could not find container \"18e1c9907ac4c4d1dcb3a430d9a8e869da48df7cd6688e69849d384f835eaa09\": container with ID starting with 18e1c9907ac4c4d1dcb3a430d9a8e869da48df7cd6688e69849d384f835eaa09 not found: ID does not exist" Oct 30 01:30:06.031427 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.031373 5300 scope.go:122] "RemoveContainer" containerID="89f9f8e5b2a4b26e049efc989b8682bf971ac3b6d804e576f57651de3a636a97" Oct 30 01:30:06.046973 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.046947 5300 scope.go:122] "RemoveContainer" containerID="89f9f8e5b2a4b26e049efc989b8682bf971ac3b6d804e576f57651de3a636a97" Oct 30 01:30:06.047999 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:30:06.047965 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89f9f8e5b2a4b26e049efc989b8682bf971ac3b6d804e576f57651de3a636a97\": container with ID starting with 89f9f8e5b2a4b26e049efc989b8682bf971ac3b6d804e576f57651de3a636a97 not found: ID does not exist" containerID="89f9f8e5b2a4b26e049efc989b8682bf971ac3b6d804e576f57651de3a636a97" Oct 30 01:30:06.048188 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.048008 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89f9f8e5b2a4b26e049efc989b8682bf971ac3b6d804e576f57651de3a636a97"} err="failed to get container status \"89f9f8e5b2a4b26e049efc989b8682bf971ac3b6d804e576f57651de3a636a97\": rpc error: code = NotFound desc = could not find container \"89f9f8e5b2a4b26e049efc989b8682bf971ac3b6d804e576f57651de3a636a97\": container with ID starting with 89f9f8e5b2a4b26e049efc989b8682bf971ac3b6d804e576f57651de3a636a97 not found: ID does not exist" Oct 30 01:30:06.200754 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.200720 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg"] Oct 30 01:30:06.208158 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.208132 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c104b8/premium-w1-c104b8-kserve-6d469c76d7-pvgrg"] Oct 30 01:30:06.254808 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.254776 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q"] Oct 30 01:30:06.264402 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.264366 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c104b8/distinct-w1-c104b8-kserve-594d959c46-9n76q"] Oct 30 01:30:06.303981 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.303946 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7"] Oct 30 01:30:06.313280 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.313239 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-29145c-677554878d-d2nx7"] Oct 30 01:30:06.628116 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.628080 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb26b84d-qgthh"] Oct 30 01:30:06.628993 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.628962 5300 cpu_manager.go:414] "containerMap: removing container" podUID="83ca74f3-ea7d-46b6-b954-f681f39ba286" containerName="payload-processing" Oct 30 01:30:06.628993 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.628989 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83ca74f3-ea7d-46b6-b954-f681f39ba286" containerName="payload-processing" Oct 30 01:30:06.629206 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.629019 5300 cpu_manager.go:414] "containerMap: removing container" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" Oct 30 01:30:06.629206 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.629027 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" Oct 30 01:30:06.629206 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.629056 5300 cpu_manager.go:414] "containerMap: removing container" podUID="d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" containerName="main" Oct 30 01:30:06.629206 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.629063 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" containerName="main" Oct 30 01:30:06.629353 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.629212 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="83ca74f3-ea7d-46b6-b954-f681f39ba286" containerName="payload-processing" Oct 30 01:30:06.629353 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.629228 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" containerName="main" Oct 30 01:30:06.629353 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.629243 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" containerName="main" Oct 30 01:30:06.634798 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.634500 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26b84d-qgthh" Oct 30 01:30:06.664593 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.664560 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb26b84d-qgthh"] Oct 30 01:30:06.732227 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.732196 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:30:06.769120 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.769087 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0577cfbe-a5b1-4408-af05-ec051233cb19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb26b84d-qgthh\" (UID: \"0577cfbe-a5b1-4408-af05-ec051233cb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26b84d-qgthh" Oct 30 01:30:06.769303 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.769278 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7bgq\" (UniqueName: \"kubernetes.io/projected/0577cfbe-a5b1-4408-af05-ec051233cb19-kube-api-access-f7bgq\") pod \"maas-api-revoke-sub-fb26b84d-qgthh\" (UID: \"0577cfbe-a5b1-4408-af05-ec051233cb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26b84d-qgthh" Oct 30 01:30:06.769365 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.769342 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0577cfbe-a5b1-4408-af05-ec051233cb19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb26b84d-qgthh\" (UID: \"0577cfbe-a5b1-4408-af05-ec051233cb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26b84d-qgthh" Oct 30 01:30:06.870855 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.870812 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0577cfbe-a5b1-4408-af05-ec051233cb19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb26b84d-qgthh\" (UID: \"0577cfbe-a5b1-4408-af05-ec051233cb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26b84d-qgthh" Oct 30 01:30:06.871060 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.871006 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7bgq\" (UniqueName: \"kubernetes.io/projected/0577cfbe-a5b1-4408-af05-ec051233cb19-kube-api-access-f7bgq\") pod \"maas-api-revoke-sub-fb26b84d-qgthh\" (UID: \"0577cfbe-a5b1-4408-af05-ec051233cb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26b84d-qgthh" Oct 30 01:30:06.871144 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.871080 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0577cfbe-a5b1-4408-af05-ec051233cb19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb26b84d-qgthh\" (UID: \"0577cfbe-a5b1-4408-af05-ec051233cb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26b84d-qgthh" Oct 30 01:30:06.871722 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.871697 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0577cfbe-a5b1-4408-af05-ec051233cb19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb26b84d-qgthh\" (UID: \"0577cfbe-a5b1-4408-af05-ec051233cb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26b84d-qgthh" Oct 30 01:30:06.873336 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.873314 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0577cfbe-a5b1-4408-af05-ec051233cb19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb26b84d-qgthh\" (UID: \"0577cfbe-a5b1-4408-af05-ec051233cb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26b84d-qgthh" Oct 30 01:30:06.894030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.893970 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7bgq\" (UniqueName: \"kubernetes.io/projected/0577cfbe-a5b1-4408-af05-ec051233cb19-kube-api-access-f7bgq\") pod \"maas-api-revoke-sub-fb26b84d-qgthh\" (UID: \"0577cfbe-a5b1-4408-af05-ec051233cb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26b84d-qgthh" Oct 30 01:30:06.954995 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:06.954961 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26b84d-qgthh" Oct 30 01:30:07.034107 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:07.033944 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:07.272781 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:07.272742 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb26b84d-qgthh"] Oct 30 01:30:07.274423 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:30:07.274384 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0577cfbe_a5b1_4408_af05_ec051233cb19.slice/crio-2bf6d6b7efc52a0e6edd490aa5987508e825e91e0c26f935be82789e863afcce WatchSource:0}: Error finding container 2bf6d6b7efc52a0e6edd490aa5987508e825e91e0c26f935be82789e863afcce: Status 404 returned error can't find the container with id 2bf6d6b7efc52a0e6edd490aa5987508e825e91e0c26f935be82789e863afcce Oct 30 01:30:07.614073 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:07.614042 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:07.779100 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:07.779014 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:07.821716 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:07.821684 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83ca74f3-ea7d-46b6-b954-f681f39ba286" path="/var/lib/kubelet/pods/83ca74f3-ea7d-46b6-b954-f681f39ba286/volumes" Oct 30 01:30:07.822210 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:07.822194 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9" path="/var/lib/kubelet/pods/a4df2fe3-8896-48bf-afdd-bdf13ca5d4a9/volumes" Oct 30 01:30:07.822614 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:07.822602 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b" path="/var/lib/kubelet/pods/d4eb07a0-4a32-4091-8db0-3d0a2f9bab8b/volumes" Oct 30 01:30:08.011289 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:08.011240 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:08.057818 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:08.057678 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26b84d-qgthh" event={"ID":"0577cfbe-a5b1-4408-af05-ec051233cb19","Type":"ContainerStarted","Data":"66682d9e2fecb793e1a51c95ee72f32db9644c49781bedd0c07c44a8320c7a0a"} Oct 30 01:30:08.057818 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:08.057746 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26b84d-qgthh" event={"ID":"0577cfbe-a5b1-4408-af05-ec051233cb19","Type":"ContainerStarted","Data":"2bf6d6b7efc52a0e6edd490aa5987508e825e91e0c26f935be82789e863afcce"} Oct 30 01:30:08.118799 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:08.115862 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26b84d-qgthh" podStartSLOduration=2.115839018 podStartE2EDuration="2.115839018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:08.114254251 +0000 UTC m=+2808.430122594" watchObservedRunningTime="2026-10-30 01:30:08.115839018 +0000 UTC m=+2808.431707343" Oct 30 01:30:08.483964 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:08.483928 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:08.850933 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:08.850852 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-389c1c22-52t4q"] Oct 30 01:30:08.859801 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:08.855406 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-389c1c22-52t4q" Oct 30 01:30:08.901125 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:08.901092 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-389c1c22-52t4q"] Oct 30 01:30:09.002355 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:09.002320 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8d5304a-f4ce-472e-b78f-e5d394677bcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-389c1c22-52t4q\" (UID: \"a8d5304a-f4ce-472e-b78f-e5d394677bcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-389c1c22-52t4q" Oct 30 01:30:09.002551 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:09.002457 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8d5304a-f4ce-472e-b78f-e5d394677bcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-389c1c22-52t4q\" (UID: \"a8d5304a-f4ce-472e-b78f-e5d394677bcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-389c1c22-52t4q" Oct 30 01:30:09.002551 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:09.002525 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92s6f\" (UniqueName: \"kubernetes.io/projected/a8d5304a-f4ce-472e-b78f-e5d394677bcd-kube-api-access-92s6f\") pod \"maas-api-revoke-sub-389c1c22-52t4q\" (UID: \"a8d5304a-f4ce-472e-b78f-e5d394677bcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-389c1c22-52t4q" Oct 30 01:30:09.066696 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:09.066661 5300 generic.go:356] "Generic (PLEG): container finished" podID="0577cfbe-a5b1-4408-af05-ec051233cb19" containerID="66682d9e2fecb793e1a51c95ee72f32db9644c49781bedd0c07c44a8320c7a0a" exitCode=0 Oct 30 01:30:09.067161 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:09.066733 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26b84d-qgthh" event={"ID":"0577cfbe-a5b1-4408-af05-ec051233cb19","Type":"ContainerDied","Data":"66682d9e2fecb793e1a51c95ee72f32db9644c49781bedd0c07c44a8320c7a0a"} Oct 30 01:30:09.104377 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:09.104292 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8d5304a-f4ce-472e-b78f-e5d394677bcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-389c1c22-52t4q\" (UID: \"a8d5304a-f4ce-472e-b78f-e5d394677bcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-389c1c22-52t4q" Oct 30 01:30:09.104544 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:09.104379 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8d5304a-f4ce-472e-b78f-e5d394677bcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-389c1c22-52t4q\" (UID: \"a8d5304a-f4ce-472e-b78f-e5d394677bcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-389c1c22-52t4q" Oct 30 01:30:09.104544 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:09.104427 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92s6f\" (UniqueName: \"kubernetes.io/projected/a8d5304a-f4ce-472e-b78f-e5d394677bcd-kube-api-access-92s6f\") pod \"maas-api-revoke-sub-389c1c22-52t4q\" (UID: \"a8d5304a-f4ce-472e-b78f-e5d394677bcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-389c1c22-52t4q" Oct 30 01:30:09.105124 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:09.105105 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8d5304a-f4ce-472e-b78f-e5d394677bcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-389c1c22-52t4q\" (UID: \"a8d5304a-f4ce-472e-b78f-e5d394677bcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-389c1c22-52t4q" Oct 30 01:30:09.106801 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:09.106774 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8d5304a-f4ce-472e-b78f-e5d394677bcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-389c1c22-52t4q\" (UID: \"a8d5304a-f4ce-472e-b78f-e5d394677bcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-389c1c22-52t4q" Oct 30 01:30:09.131147 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:09.131119 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92s6f\" (UniqueName: \"kubernetes.io/projected/a8d5304a-f4ce-472e-b78f-e5d394677bcd-kube-api-access-92s6f\") pod \"maas-api-revoke-sub-389c1c22-52t4q\" (UID: \"a8d5304a-f4ce-472e-b78f-e5d394677bcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-389c1c22-52t4q" Oct 30 01:30:09.172505 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:09.172472 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-389c1c22-52t4q" Oct 30 01:30:09.259806 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:09.255746 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:09.492237 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:09.492199 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-389c1c22-52t4q"] Oct 30 01:30:09.493093 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:30:09.493059 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d5304a_f4ce_472e_b78f_e5d394677bcd.slice/crio-23319864c41e054cc648a34c75eb459a8b24df9592847e958bda9f085f08221c WatchSource:0}: Error finding container 23319864c41e054cc648a34c75eb459a8b24df9592847e958bda9f085f08221c: Status 404 returned error can't find the container with id 23319864c41e054cc648a34c75eb459a8b24df9592847e958bda9f085f08221c Oct 30 01:30:10.074586 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:10.074547 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-389c1c22-52t4q" event={"ID":"a8d5304a-f4ce-472e-b78f-e5d394677bcd","Type":"ContainerStarted","Data":"2c59ea7c4af5aaf5a97915ef4247ce2d5de2ed800c8c1b21ba0b4c1df5ba72b5"} Oct 30 01:30:10.075061 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:10.074609 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-389c1c22-52t4q" event={"ID":"a8d5304a-f4ce-472e-b78f-e5d394677bcd","Type":"ContainerStarted","Data":"23319864c41e054cc648a34c75eb459a8b24df9592847e958bda9f085f08221c"} Oct 30 01:30:10.120611 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:10.120503 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-389c1c22-52t4q" podStartSLOduration=2.120482489 podStartE2EDuration="2.120482489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:10.118578152 +0000 UTC m=+2810.434446478" watchObservedRunningTime="2026-10-30 01:30:10.120482489 +0000 UTC m=+2810.436350815" Oct 30 01:30:10.234131 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:10.234105 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26b84d-qgthh" Oct 30 01:30:10.322186 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:10.322152 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0577cfbe-a5b1-4408-af05-ec051233cb19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0577cfbe-a5b1-4408-af05-ec051233cb19-maas-api-xks-ca\") pod \"0577cfbe-a5b1-4408-af05-ec051233cb19\" (UID: \"0577cfbe-a5b1-4408-af05-ec051233cb19\") " Oct 30 01:30:10.322380 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:10.322315 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0577cfbe-a5b1-4408-af05-ec051233cb19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0577cfbe-a5b1-4408-af05-ec051233cb19-maas-api-openshift-ca\") pod \"0577cfbe-a5b1-4408-af05-ec051233cb19\" (UID: \"0577cfbe-a5b1-4408-af05-ec051233cb19\") " Oct 30 01:30:10.322380 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:10.322370 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0577cfbe-a5b1-4408-af05-ec051233cb19-kube-api-access-f7bgq\" (UniqueName: \"kubernetes.io/projected/0577cfbe-a5b1-4408-af05-ec051233cb19-kube-api-access-f7bgq\") pod \"0577cfbe-a5b1-4408-af05-ec051233cb19\" (UID: \"0577cfbe-a5b1-4408-af05-ec051233cb19\") " Oct 30 01:30:10.322741 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:10.322713 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0577cfbe-a5b1-4408-af05-ec051233cb19-maas-api-openshift-ca" pod "0577cfbe-a5b1-4408-af05-ec051233cb19" (UID: "0577cfbe-a5b1-4408-af05-ec051233cb19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:10.323149 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:10.323124 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0577cfbe-a5b1-4408-af05-ec051233cb19-maas-api-openshift-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:10.325021 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:10.324481 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0577cfbe-a5b1-4408-af05-ec051233cb19-maas-api-xks-ca" pod "0577cfbe-a5b1-4408-af05-ec051233cb19" (UID: "0577cfbe-a5b1-4408-af05-ec051233cb19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:10.325414 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:10.325386 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0577cfbe-a5b1-4408-af05-ec051233cb19-kube-api-access-f7bgq" pod "0577cfbe-a5b1-4408-af05-ec051233cb19" (UID: "0577cfbe-a5b1-4408-af05-ec051233cb19"). InnerVolumeSpecName "kube-api-access-f7bgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:10.425385 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:10.425351 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7bgq\" (UniqueName: \"kubernetes.io/projected/0577cfbe-a5b1-4408-af05-ec051233cb19-kube-api-access-f7bgq\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:10.425385 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:10.425386 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0577cfbe-a5b1-4408-af05-ec051233cb19-maas-api-xks-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:10.759690 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:10.759662 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:10.954270 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:10.954236 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:11.051636 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:11.051536 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:11.084254 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:11.084209 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26b84d-qgthh" Oct 30 01:30:11.084754 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:11.084204 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26b84d-qgthh" event={"ID":"0577cfbe-a5b1-4408-af05-ec051233cb19","Type":"ContainerDied","Data":"2bf6d6b7efc52a0e6edd490aa5987508e825e91e0c26f935be82789e863afcce"} Oct 30 01:30:11.084935 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:11.084812 5300 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bf6d6b7efc52a0e6edd490aa5987508e825e91e0c26f935be82789e863afcce" Oct 30 01:30:11.086171 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:11.086143 5300 generic.go:356] "Generic (PLEG): container finished" podID="a8d5304a-f4ce-472e-b78f-e5d394677bcd" containerID="2c59ea7c4af5aaf5a97915ef4247ce2d5de2ed800c8c1b21ba0b4c1df5ba72b5" exitCode=0 Oct 30 01:30:11.086288 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:11.086212 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-389c1c22-52t4q" event={"ID":"a8d5304a-f4ce-472e-b78f-e5d394677bcd","Type":"ContainerDied","Data":"2c59ea7c4af5aaf5a97915ef4247ce2d5de2ed800c8c1b21ba0b4c1df5ba72b5"} Oct 30 01:30:11.365842 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:11.362164 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:11.497645 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:11.497611 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:11.600154 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:11.600124 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:11.726965 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:11.726938 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:12.120243 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:12.120207 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:12.245407 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:12.245378 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-389c1c22-52t4q" Oct 30 01:30:12.353295 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:12.353206 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8d5304a-f4ce-472e-b78f-e5d394677bcd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8d5304a-f4ce-472e-b78f-e5d394677bcd-maas-api-xks-ca\") pod \"a8d5304a-f4ce-472e-b78f-e5d394677bcd\" (UID: \"a8d5304a-f4ce-472e-b78f-e5d394677bcd\") " Oct 30 01:30:12.353437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:12.353349 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8d5304a-f4ce-472e-b78f-e5d394677bcd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8d5304a-f4ce-472e-b78f-e5d394677bcd-maas-api-openshift-ca\") pod \"a8d5304a-f4ce-472e-b78f-e5d394677bcd\" (UID: \"a8d5304a-f4ce-472e-b78f-e5d394677bcd\") " Oct 30 01:30:12.353437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:12.353393 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8d5304a-f4ce-472e-b78f-e5d394677bcd-kube-api-access-92s6f\" (UniqueName: \"kubernetes.io/projected/a8d5304a-f4ce-472e-b78f-e5d394677bcd-kube-api-access-92s6f\") pod \"a8d5304a-f4ce-472e-b78f-e5d394677bcd\" (UID: \"a8d5304a-f4ce-472e-b78f-e5d394677bcd\") " Oct 30 01:30:12.353732 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:12.353702 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d5304a-f4ce-472e-b78f-e5d394677bcd-maas-api-openshift-ca" pod "a8d5304a-f4ce-472e-b78f-e5d394677bcd" (UID: "a8d5304a-f4ce-472e-b78f-e5d394677bcd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:12.355435 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:12.355410 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8d5304a-f4ce-472e-b78f-e5d394677bcd-maas-api-xks-ca" pod "a8d5304a-f4ce-472e-b78f-e5d394677bcd" (UID: "a8d5304a-f4ce-472e-b78f-e5d394677bcd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:12.355600 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:12.355579 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d5304a-f4ce-472e-b78f-e5d394677bcd-kube-api-access-92s6f" pod "a8d5304a-f4ce-472e-b78f-e5d394677bcd" (UID: "a8d5304a-f4ce-472e-b78f-e5d394677bcd"). InnerVolumeSpecName "kube-api-access-92s6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:12.455246 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:12.455206 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8d5304a-f4ce-472e-b78f-e5d394677bcd-maas-api-xks-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:12.455246 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:12.455240 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8d5304a-f4ce-472e-b78f-e5d394677bcd-maas-api-openshift-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:12.455246 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:12.455250 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92s6f\" (UniqueName: \"kubernetes.io/projected/a8d5304a-f4ce-472e-b78f-e5d394677bcd-kube-api-access-92s6f\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:13.117791 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:13.113550 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-389c1c22-52t4q" event={"ID":"a8d5304a-f4ce-472e-b78f-e5d394677bcd","Type":"ContainerDied","Data":"23319864c41e054cc648a34c75eb459a8b24df9592847e958bda9f085f08221c"} Oct 30 01:30:13.117791 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:13.113614 5300 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23319864c41e054cc648a34c75eb459a8b24df9592847e958bda9f085f08221c" Oct 30 01:30:13.117791 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:13.113683 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-389c1c22-52t4q" Oct 30 01:30:13.405723 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:13.405621 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:13.418439 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:13.418399 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q"] Oct 30 01:30:13.577089 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:13.576051 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:15.345963 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:15.345927 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:15.813652 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:15.813623 5300 scope.go:122] "RemoveContainer" containerID="fca08f3cef1ddb9b13f8e82c595abd33410dad0b2eb32507a5088a06c5658848" Oct 30 01:30:15.813861 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:15.813834 5300 scope.go:122] "RemoveContainer" containerID="19043f6c80ca637f59e2078af8b185572aa410ac7a26fe8f74977b9b2cfe8a83" Oct 30 01:30:16.813925 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:16.813886 5300 scope.go:122] "RemoveContainer" containerID="da168eaca863c9deb7e5b3bc7d821fc5ddf5f58430684a6949be87fc81103b8b" Oct 30 01:30:17.148848 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:17.147953 5300 generic.go:356] "Generic (PLEG): container finished" podID="e15f0895-ce3b-401d-af86-68670b59de95" containerID="e1b1ce66790bf46d8bdf70e48059c2919b9ac795579f6d39cda236d9e4747106" exitCode=60 Oct 30 01:30:17.148848 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:17.148176 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc" event={"ID":"e15f0895-ce3b-401d-af86-68670b59de95","Type":"ContainerDied","Data":"e1b1ce66790bf46d8bdf70e48059c2919b9ac795579f6d39cda236d9e4747106"} Oct 30 01:30:17.148848 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:17.148242 5300 scope.go:122] "RemoveContainer" containerID="19043f6c80ca637f59e2078af8b185572aa410ac7a26fe8f74977b9b2cfe8a83" Oct 30 01:30:17.148848 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:17.148714 5300 scope.go:122] "RemoveContainer" containerID="e1b1ce66790bf46d8bdf70e48059c2919b9ac795579f6d39cda236d9e4747106" Oct 30 01:30:17.149151 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:30:17.149124 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc_odh-ai-gateway-infra(e15f0895-ce3b-401d-af86-68670b59de95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc" podUID="e15f0895-ce3b-401d-af86-68670b59de95" Oct 30 01:30:17.150539 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:17.150516 5300 generic.go:356] "Generic (PLEG): container finished" podID="2c39939a-6ce7-4527-b066-ce41e01d74d4" containerID="233db0d537b35790a727d5a728dbb3ea503748d399413bd5bdb510f597ecae1e" exitCode=60 Oct 30 01:30:17.150656 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:17.150589 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv" event={"ID":"2c39939a-6ce7-4527-b066-ce41e01d74d4","Type":"ContainerDied","Data":"233db0d537b35790a727d5a728dbb3ea503748d399413bd5bdb510f597ecae1e"} Oct 30 01:30:17.151065 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:17.151042 5300 scope.go:122] "RemoveContainer" containerID="233db0d537b35790a727d5a728dbb3ea503748d399413bd5bdb510f597ecae1e" Oct 30 01:30:17.151467 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:30:17.151441 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv_odh-ai-gateway-infra(2c39939a-6ce7-4527-b066-ce41e01d74d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv" podUID="2c39939a-6ce7-4527-b066-ce41e01d74d4" Oct 30 01:30:17.155292 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:17.155267 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-mmvfp" event={"ID":"c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a","Type":"ContainerStarted","Data":"b781d2b18df9fb01012de13ff0c61b375dea6ae5972c89f99f01a28349294c13"} Oct 30 01:30:17.165492 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:17.165468 5300 scope.go:122] "RemoveContainer" containerID="fca08f3cef1ddb9b13f8e82c595abd33410dad0b2eb32507a5088a06c5658848" Oct 30 01:30:17.662250 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:17.662213 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:17.821355 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:17.821326 5300 scope.go:122] "RemoveContainer" containerID="179085c3ea0d825f4c6a0d259b9f8d1d89580aeea79301d4bbecc03188ac6f96" Oct 30 01:30:18.168595 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:18.168562 5300 generic.go:356] "Generic (PLEG): container finished" podID="c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a" containerID="b781d2b18df9fb01012de13ff0c61b375dea6ae5972c89f99f01a28349294c13" exitCode=60 Oct 30 01:30:18.168748 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:18.168636 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-mmvfp" event={"ID":"c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a","Type":"ContainerDied","Data":"b781d2b18df9fb01012de13ff0c61b375dea6ae5972c89f99f01a28349294c13"} Oct 30 01:30:18.168748 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:18.168701 5300 scope.go:122] "RemoveContainer" containerID="da168eaca863c9deb7e5b3bc7d821fc5ddf5f58430684a6949be87fc81103b8b" Oct 30 01:30:18.169168 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:18.169143 5300 scope.go:122] "RemoveContainer" containerID="b781d2b18df9fb01012de13ff0c61b375dea6ae5972c89f99f01a28349294c13" Oct 30 01:30:18.169485 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:30:18.169461 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888730-mmvfp_odh-ai-gateway-infra(c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-mmvfp" podUID="c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a" Oct 30 01:30:18.170643 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:18.170623 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92" event={"ID":"789a14f0-e4ff-4e2e-b587-3ed5cca21d54","Type":"ContainerStarted","Data":"03a0eebf5703333e37bd44a1a6b447acac7396a22191beac364e024a1099ce48"} Oct 30 01:30:18.238811 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:18.238777 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc"] Oct 30 01:30:18.293099 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:18.293011 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv"] Oct 30 01:30:18.356302 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:18.356201 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-mmvfp"] Oct 30 01:30:18.464178 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:18.464146 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc" Oct 30 01:30:18.527615 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:18.527590 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv" Oct 30 01:30:18.566355 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:18.566267 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c39939a-6ce7-4527-b066-ce41e01d74d4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c39939a-6ce7-4527-b066-ce41e01d74d4-maas-api-service-ca\") pod \"2c39939a-6ce7-4527-b066-ce41e01d74d4\" (UID: \"2c39939a-6ce7-4527-b066-ce41e01d74d4\") " Oct 30 01:30:18.566355 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:18.566322 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c39939a-6ce7-4527-b066-ce41e01d74d4-kube-api-access-fn5wh\" (UniqueName: \"kubernetes.io/projected/2c39939a-6ce7-4527-b066-ce41e01d74d4-kube-api-access-fn5wh\") pod \"2c39939a-6ce7-4527-b066-ce41e01d74d4\" (UID: \"2c39939a-6ce7-4527-b066-ce41e01d74d4\") " Oct 30 01:30:18.566593 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:18.566446 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e15f0895-ce3b-401d-af86-68670b59de95-kube-api-access-4g9rz\" (UniqueName: \"kubernetes.io/projected/e15f0895-ce3b-401d-af86-68670b59de95-kube-api-access-4g9rz\") pod \"e15f0895-ce3b-401d-af86-68670b59de95\" (UID: \"e15f0895-ce3b-401d-af86-68670b59de95\") " Oct 30 01:30:18.566593 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:18.566505 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e15f0895-ce3b-401d-af86-68670b59de95-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e15f0895-ce3b-401d-af86-68670b59de95-maas-api-service-ca\") pod \"e15f0895-ce3b-401d-af86-68670b59de95\" (UID: \"e15f0895-ce3b-401d-af86-68670b59de95\") " Oct 30 01:30:18.566736 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:18.566701 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c39939a-6ce7-4527-b066-ce41e01d74d4-maas-api-service-ca" pod "2c39939a-6ce7-4527-b066-ce41e01d74d4" (UID: "2c39939a-6ce7-4527-b066-ce41e01d74d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:18.567152 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:18.567126 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c39939a-6ce7-4527-b066-ce41e01d74d4-maas-api-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:18.567275 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:18.567151 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e15f0895-ce3b-401d-af86-68670b59de95-maas-api-service-ca" pod "e15f0895-ce3b-401d-af86-68670b59de95" (UID: "e15f0895-ce3b-401d-af86-68670b59de95"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:18.568748 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:18.568722 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e15f0895-ce3b-401d-af86-68670b59de95-kube-api-access-4g9rz" pod "e15f0895-ce3b-401d-af86-68670b59de95" (UID: "e15f0895-ce3b-401d-af86-68670b59de95"). InnerVolumeSpecName "kube-api-access-4g9rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:18.568993 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:18.568972 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c39939a-6ce7-4527-b066-ce41e01d74d4-kube-api-access-fn5wh" pod "2c39939a-6ce7-4527-b066-ce41e01d74d4" (UID: "2c39939a-6ce7-4527-b066-ce41e01d74d4"). InnerVolumeSpecName "kube-api-access-fn5wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:18.668166 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:18.668130 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn5wh\" (UniqueName: \"kubernetes.io/projected/2c39939a-6ce7-4527-b066-ce41e01d74d4-kube-api-access-fn5wh\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:18.668166 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:18.668162 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g9rz\" (UniqueName: \"kubernetes.io/projected/e15f0895-ce3b-401d-af86-68670b59de95-kube-api-access-4g9rz\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:18.668373 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:18.668179 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e15f0895-ce3b-401d-af86-68670b59de95-maas-api-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:19.156549 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.156444 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb"] Oct 30 01:30:19.179216 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.179186 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc" Oct 30 01:30:19.179371 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.179227 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc" event={"ID":"e15f0895-ce3b-401d-af86-68670b59de95","Type":"ContainerDied","Data":"da6f5d0b858631bd4133e4be61496529f0563ee2fa44dc6f1516e0a27f15a6da"} Oct 30 01:30:19.179371 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.179290 5300 scope.go:122] "RemoveContainer" containerID="e1b1ce66790bf46d8bdf70e48059c2919b9ac795579f6d39cda236d9e4747106" Oct 30 01:30:19.180878 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.180858 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv" Oct 30 01:30:19.181006 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.180860 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv" event={"ID":"2c39939a-6ce7-4527-b066-ce41e01d74d4","Type":"ContainerDied","Data":"7b4dc6d8b1101dab352738970b91dca9a34a4c9f345656998e3a7efd28949e4f"} Oct 30 01:30:19.185031 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.185007 5300 generic.go:356] "Generic (PLEG): container finished" podID="789a14f0-e4ff-4e2e-b587-3ed5cca21d54" containerID="03a0eebf5703333e37bd44a1a6b447acac7396a22191beac364e024a1099ce48" exitCode=60 Oct 30 01:30:19.185149 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.185085 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92" event={"ID":"789a14f0-e4ff-4e2e-b587-3ed5cca21d54","Type":"ContainerDied","Data":"03a0eebf5703333e37bd44a1a6b447acac7396a22191beac364e024a1099ce48"} Oct 30 01:30:19.185546 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.185522 5300 scope.go:122] "RemoveContainer" containerID="03a0eebf5703333e37bd44a1a6b447acac7396a22191beac364e024a1099ce48" Oct 30 01:30:19.185899 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:30:19.185869 5300 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92_odh-ai-gateway-infra(789a14f0-e4ff-4e2e-b587-3ed5cca21d54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92" podUID="789a14f0-e4ff-4e2e-b587-3ed5cca21d54" Oct 30 01:30:19.191562 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.191542 5300 scope.go:122] "RemoveContainer" containerID="233db0d537b35790a727d5a728dbb3ea503748d399413bd5bdb510f597ecae1e" Oct 30 01:30:19.202878 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.202852 5300 scope.go:122] "RemoveContainer" containerID="179085c3ea0d825f4c6a0d259b9f8d1d89580aeea79301d4bbecc03188ac6f96" Oct 30 01:30:19.203615 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.203594 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm"] Oct 30 01:30:19.291299 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.270010 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc"] Oct 30 01:30:19.293427 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.293395 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e92648-29888730-n9tqc"] Oct 30 01:30:19.297260 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.297233 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92"] Oct 30 01:30:19.334806 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.334596 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv"] Oct 30 01:30:19.339261 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.338852 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:19.343243 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.343212 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-eaf753-29888730-xg7sv"] Oct 30 01:30:19.372522 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.372495 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-mmvfp" Oct 30 01:30:19.468268 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.468177 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:19.480400 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.480368 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a-kube-api-access-bl2n8\" (UniqueName: \"kubernetes.io/projected/c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a-kube-api-access-bl2n8\") pod \"c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a\" (UID: \"c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a\") " Oct 30 01:30:19.480589 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.480576 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a-maas-api-service-ca\") pod \"c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a\" (UID: \"c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a\") " Oct 30 01:30:19.480953 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.480927 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a-maas-api-service-ca" pod "c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a" (UID: "c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:19.481054 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.481036 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a-maas-api-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:19.482654 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.482628 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a-kube-api-access-bl2n8" pod "c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a" (UID: "c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a"). InnerVolumeSpecName "kube-api-access-bl2n8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:19.582899 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.582865 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl2n8\" (UniqueName: \"kubernetes.io/projected/c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a-kube-api-access-bl2n8\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:19.823103 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.823073 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c39939a-6ce7-4527-b066-ce41e01d74d4" path="/var/lib/kubelet/pods/2c39939a-6ce7-4527-b066-ce41e01d74d4/volumes" Oct 30 01:30:19.823561 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:19.823546 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e15f0895-ce3b-401d-af86-68670b59de95" path="/var/lib/kubelet/pods/e15f0895-ce3b-401d-af86-68670b59de95/volumes" Oct 30 01:30:20.150018 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:20.149838 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:20.195134 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:20.195107 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-mmvfp" Oct 30 01:30:20.195134 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:20.195117 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888730-mmvfp" event={"ID":"c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a","Type":"ContainerDied","Data":"509753fbfa589748d953b39578bc7ad627c417dccb92e3e46f723fef89004ef0"} Oct 30 01:30:20.195954 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:20.195181 5300 scope.go:122] "RemoveContainer" containerID="b781d2b18df9fb01012de13ff0c61b375dea6ae5972c89f99f01a28349294c13" Oct 30 01:30:20.308464 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:20.308433 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-mmvfp"] Oct 30 01:30:20.320244 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:20.320209 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888730-mmvfp"] Oct 30 01:30:20.365997 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:20.365972 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92" Oct 30 01:30:20.495752 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:20.495720 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789a14f0-e4ff-4e2e-b587-3ed5cca21d54-kube-api-access-9d4q8\" (UniqueName: \"kubernetes.io/projected/789a14f0-e4ff-4e2e-b587-3ed5cca21d54-kube-api-access-9d4q8\") pod \"789a14f0-e4ff-4e2e-b587-3ed5cca21d54\" (UID: \"789a14f0-e4ff-4e2e-b587-3ed5cca21d54\") " Oct 30 01:30:20.495952 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:20.495855 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/789a14f0-e4ff-4e2e-b587-3ed5cca21d54-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/789a14f0-e4ff-4e2e-b587-3ed5cca21d54-maas-api-service-ca\") pod \"789a14f0-e4ff-4e2e-b587-3ed5cca21d54\" (UID: \"789a14f0-e4ff-4e2e-b587-3ed5cca21d54\") " Oct 30 01:30:20.496210 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:20.496183 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789a14f0-e4ff-4e2e-b587-3ed5cca21d54-maas-api-service-ca" pod "789a14f0-e4ff-4e2e-b587-3ed5cca21d54" (UID: "789a14f0-e4ff-4e2e-b587-3ed5cca21d54"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:20.496432 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:20.496419 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/789a14f0-e4ff-4e2e-b587-3ed5cca21d54-maas-api-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:20.497783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:20.497742 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789a14f0-e4ff-4e2e-b587-3ed5cca21d54-kube-api-access-9d4q8" pod "789a14f0-e4ff-4e2e-b587-3ed5cca21d54" (UID: "789a14f0-e4ff-4e2e-b587-3ed5cca21d54"). InnerVolumeSpecName "kube-api-access-9d4q8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:20.597335 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:20.597300 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d4q8\" (UniqueName: \"kubernetes.io/projected/789a14f0-e4ff-4e2e-b587-3ed5cca21d54-kube-api-access-9d4q8\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:20.821057 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:20.820961 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:20.925998 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:20.925962 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:21.193024 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:21.192939 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d"] Oct 30 01:30:21.193241 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:21.193207 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" podUID="85d29953-787a-49a2-87d5-1514f1c425ee" containerName="istio-proxy" containerID="cri-o://a12c568c8faf5863370778a422fd81d0ad39bbb2cad6c9c404a8b0fdf8a94bcc" gracePeriod=30 Oct 30 01:30:21.210173 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:21.210143 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92" event={"ID":"789a14f0-e4ff-4e2e-b587-3ed5cca21d54","Type":"ContainerDied","Data":"5463f57e5a6675a1b0fd82196fcf49c908ae28493dfe9e3bd2c2f43d6768814a"} Oct 30 01:30:21.210616 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:21.210189 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92" Oct 30 01:30:21.210616 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:21.210200 5300 scope.go:122] "RemoveContainer" containerID="03a0eebf5703333e37bd44a1a6b447acac7396a22191beac364e024a1099ce48" Oct 30 01:30:21.253008 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:21.252974 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:21.372926 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:21.372889 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92"] Oct 30 01:30:21.384895 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:21.383389 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fd3281-29888730-jjv92"] Oct 30 01:30:21.593906 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:30:21.593858 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:30:21.822285 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:21.822255 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="789a14f0-e4ff-4e2e-b587-3ed5cca21d54" path="/var/lib/kubelet/pods/789a14f0-e4ff-4e2e-b587-3ed5cca21d54/volumes" Oct 30 01:30:21.822936 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:21.822914 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a" path="/var/lib/kubelet/pods/c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a/volumes" Oct 30 01:30:21.834077 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:21.834048 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr"] Oct 30 01:30:21.834304 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:21.834280 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr" podUID="fdd0a075-d5cf-4982-9623-c5432a308de6" containerName="maas-api" containerID="cri-o://764a2720e72cb36fb2ee7bc1d3074f71c8015bb427a8aec55b9847794d04ea3f" gracePeriod=30 Oct 30 01:30:21.907732 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:21.907699 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:22.199640 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:22.199613 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr" Oct 30 01:30:22.217799 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:22.217741 5300 generic.go:356] "Generic (PLEG): container finished" podID="fdd0a075-d5cf-4982-9623-c5432a308de6" containerID="764a2720e72cb36fb2ee7bc1d3074f71c8015bb427a8aec55b9847794d04ea3f" exitCode=0 Oct 30 01:30:22.217799 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:22.217789 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr" event={"ID":"fdd0a075-d5cf-4982-9623-c5432a308de6","Type":"ContainerDied","Data":"764a2720e72cb36fb2ee7bc1d3074f71c8015bb427a8aec55b9847794d04ea3f"} Oct 30 01:30:22.217799 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:22.217805 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr" Oct 30 01:30:22.218416 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:22.217893 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr" event={"ID":"fdd0a075-d5cf-4982-9623-c5432a308de6","Type":"ContainerDied","Data":"f380c59eb34dd58c235f5696ea14e5457d6b9e1b7cb4e67bddf64796270587bc"} Oct 30 01:30:22.218416 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:22.217928 5300 scope.go:122] "RemoveContainer" containerID="764a2720e72cb36fb2ee7bc1d3074f71c8015bb427a8aec55b9847794d04ea3f" Oct 30 01:30:22.227606 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:22.227587 5300 scope.go:122] "RemoveContainer" containerID="764a2720e72cb36fb2ee7bc1d3074f71c8015bb427a8aec55b9847794d04ea3f" Oct 30 01:30:22.228132 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:30:22.228105 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"764a2720e72cb36fb2ee7bc1d3074f71c8015bb427a8aec55b9847794d04ea3f\": container with ID starting with 764a2720e72cb36fb2ee7bc1d3074f71c8015bb427a8aec55b9847794d04ea3f not found: ID does not exist" containerID="764a2720e72cb36fb2ee7bc1d3074f71c8015bb427a8aec55b9847794d04ea3f" Oct 30 01:30:22.228207 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:22.228142 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"764a2720e72cb36fb2ee7bc1d3074f71c8015bb427a8aec55b9847794d04ea3f"} err="failed to get container status \"764a2720e72cb36fb2ee7bc1d3074f71c8015bb427a8aec55b9847794d04ea3f\": rpc error: code = NotFound desc = could not find container \"764a2720e72cb36fb2ee7bc1d3074f71c8015bb427a8aec55b9847794d04ea3f\": container with ID starting with 764a2720e72cb36fb2ee7bc1d3074f71c8015bb427a8aec55b9847794d04ea3f not found: ID does not exist" Oct 30 01:30:22.393912 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:22.387605 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdd0a075-d5cf-4982-9623-c5432a308de6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdd0a075-d5cf-4982-9623-c5432a308de6-openshift-service-ca\") pod \"fdd0a075-d5cf-4982-9623-c5432a308de6\" (UID: \"fdd0a075-d5cf-4982-9623-c5432a308de6\") " Oct 30 01:30:22.393912 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:22.387674 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdd0a075-d5cf-4982-9623-c5432a308de6-kube-api-access-4nmqx\" (UniqueName: \"kubernetes.io/projected/fdd0a075-d5cf-4982-9623-c5432a308de6-kube-api-access-4nmqx\") pod \"fdd0a075-d5cf-4982-9623-c5432a308de6\" (UID: \"fdd0a075-d5cf-4982-9623-c5432a308de6\") " Oct 30 01:30:22.393912 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:22.387748 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdd0a075-d5cf-4982-9623-c5432a308de6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd0a075-d5cf-4982-9623-c5432a308de6-metrics-certs\") pod \"fdd0a075-d5cf-4982-9623-c5432a308de6\" (UID: \"fdd0a075-d5cf-4982-9623-c5432a308de6\") " Oct 30 01:30:22.393912 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:22.387839 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdd0a075-d5cf-4982-9623-c5432a308de6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdd0a075-d5cf-4982-9623-c5432a308de6-maas-api-tls\") pod \"fdd0a075-d5cf-4982-9623-c5432a308de6\" (UID: \"fdd0a075-d5cf-4982-9623-c5432a308de6\") " Oct 30 01:30:22.393912 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:22.388649 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd0a075-d5cf-4982-9623-c5432a308de6-openshift-service-ca" pod "fdd0a075-d5cf-4982-9623-c5432a308de6" (UID: "fdd0a075-d5cf-4982-9623-c5432a308de6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:22.426831 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:22.418969 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd0a075-d5cf-4982-9623-c5432a308de6-kube-api-access-4nmqx" pod "fdd0a075-d5cf-4982-9623-c5432a308de6" (UID: "fdd0a075-d5cf-4982-9623-c5432a308de6"). InnerVolumeSpecName "kube-api-access-4nmqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:22.446785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:22.445563 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdd0a075-d5cf-4982-9623-c5432a308de6-maas-api-tls" pod "fdd0a075-d5cf-4982-9623-c5432a308de6" (UID: "fdd0a075-d5cf-4982-9623-c5432a308de6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:22.460630 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:22.458146 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdd0a075-d5cf-4982-9623-c5432a308de6-metrics-certs" pod "fdd0a075-d5cf-4982-9623-c5432a308de6" (UID: "fdd0a075-d5cf-4982-9623-c5432a308de6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:22.489372 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:22.489335 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nmqx\" (UniqueName: \"kubernetes.io/projected/fdd0a075-d5cf-4982-9623-c5432a308de6-kube-api-access-4nmqx\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:22.489372 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:22.489370 5300 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdd0a075-d5cf-4982-9623-c5432a308de6-metrics-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:22.489372 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:22.489379 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdd0a075-d5cf-4982-9623-c5432a308de6-maas-api-tls\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:22.489626 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:22.489389 5300 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdd0a075-d5cf-4982-9623-c5432a308de6-openshift-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:22.609272 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:22.609237 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr"] Oct 30 01:30:22.633560 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:22.633515 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-fd3281-546b5f9858-sl7tr"] Oct 30 01:30:22.711575 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:22.711494 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:22.907668 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:22.907637 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:23.088542 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:23.088505 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:23.383152 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:23.383067 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:23.822286 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:23.822253 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdd0a075-d5cf-4982-9623-c5432a308de6" path="/var/lib/kubelet/pods/fdd0a075-d5cf-4982-9623-c5432a308de6/volumes" Oct 30 01:30:24.157895 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:24.157785 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb" podUID="7f309af2-1f23-4295-8187-129275259d8a" containerName="payload-processing" containerID="cri-o://981208c4e05bf33db5ad768118b71b789f8794874cea061663aa65a16be50af5" gracePeriod=25 Oct 30 01:30:24.204576 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:24.204531 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm" podUID="5c19f786-50e8-4dee-97ae-c4445035f930" containerName="payload-pre-processing" containerID="cri-o://afbf1873ed4296546c649b91a42b600c578726cb07be0da8c37020648290bcdb" gracePeriod=25 Oct 30 01:30:24.791332 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:24.791304 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:26.473289 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.473264 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:30:26.543177 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.543136 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-istio-data\") pod \"85d29953-787a-49a2-87d5-1514f1c425ee\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " Oct 30 01:30:26.543352 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.543199 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85d29953-787a-49a2-87d5-1514f1c425ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d29953-787a-49a2-87d5-1514f1c425ee-istio-podinfo\") pod \"85d29953-787a-49a2-87d5-1514f1c425ee\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " Oct 30 01:30:26.543352 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.543244 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-istio-envoy\") pod \"85d29953-787a-49a2-87d5-1514f1c425ee\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " Oct 30 01:30:26.543352 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.543305 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-workload-socket\") pod \"85d29953-787a-49a2-87d5-1514f1c425ee\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " Oct 30 01:30:26.543540 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.543361 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85d29953-787a-49a2-87d5-1514f1c425ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d29953-787a-49a2-87d5-1514f1c425ee-istiod-ca-cert\") pod \"85d29953-787a-49a2-87d5-1514f1c425ee\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " Oct 30 01:30:26.543540 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.543414 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-credential-socket\") pod \"85d29953-787a-49a2-87d5-1514f1c425ee\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " Oct 30 01:30:26.543540 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.543482 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-istio-data" pod "85d29953-787a-49a2-87d5-1514f1c425ee" (UID: "85d29953-787a-49a2-87d5-1514f1c425ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:26.543540 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.543528 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85d29953-787a-49a2-87d5-1514f1c425ee-kube-api-access-br656\" (UniqueName: \"kubernetes.io/projected/85d29953-787a-49a2-87d5-1514f1c425ee-kube-api-access-br656\") pod \"85d29953-787a-49a2-87d5-1514f1c425ee\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " Oct 30 01:30:26.543709 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.543566 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85d29953-787a-49a2-87d5-1514f1c425ee-istio-token\" (UniqueName: \"kubernetes.io/projected/85d29953-787a-49a2-87d5-1514f1c425ee-istio-token\") pod \"85d29953-787a-49a2-87d5-1514f1c425ee\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " Oct 30 01:30:26.543709 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.543624 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-workload-socket" pod "85d29953-787a-49a2-87d5-1514f1c425ee" (UID: "85d29953-787a-49a2-87d5-1514f1c425ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:26.543709 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.543665 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-workload-certs\") pod \"85d29953-787a-49a2-87d5-1514f1c425ee\" (UID: \"85d29953-787a-49a2-87d5-1514f1c425ee\") " Oct 30 01:30:26.543914 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.543871 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-credential-socket" pod "85d29953-787a-49a2-87d5-1514f1c425ee" (UID: "85d29953-787a-49a2-87d5-1514f1c425ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:26.543968 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.543942 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-workload-certs" pod "85d29953-787a-49a2-87d5-1514f1c425ee" (UID: "85d29953-787a-49a2-87d5-1514f1c425ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:26.544013 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.543964 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85d29953-787a-49a2-87d5-1514f1c425ee-istiod-ca-cert" pod "85d29953-787a-49a2-87d5-1514f1c425ee" (UID: "85d29953-787a-49a2-87d5-1514f1c425ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:26.544357 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.544335 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:26.544528 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.544512 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:26.544640 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.544618 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:26.544640 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.544640 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:26.544839 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.544656 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d29953-787a-49a2-87d5-1514f1c425ee-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:26.546070 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.546047 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d29953-787a-49a2-87d5-1514f1c425ee-istio-token" pod "85d29953-787a-49a2-87d5-1514f1c425ee" (UID: "85d29953-787a-49a2-87d5-1514f1c425ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:26.546209 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.546146 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85d29953-787a-49a2-87d5-1514f1c425ee-istio-podinfo" pod "85d29953-787a-49a2-87d5-1514f1c425ee" (UID: "85d29953-787a-49a2-87d5-1514f1c425ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:30:26.546327 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.546247 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d29953-787a-49a2-87d5-1514f1c425ee-kube-api-access-br656" pod "85d29953-787a-49a2-87d5-1514f1c425ee" (UID: "85d29953-787a-49a2-87d5-1514f1c425ee"). InnerVolumeSpecName "kube-api-access-br656". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:26.546327 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.546292 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-istio-envoy" pod "85d29953-787a-49a2-87d5-1514f1c425ee" (UID: "85d29953-787a-49a2-87d5-1514f1c425ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:26.645505 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.645469 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br656\" (UniqueName: \"kubernetes.io/projected/85d29953-787a-49a2-87d5-1514f1c425ee-kube-api-access-br656\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:26.645505 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.645501 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d29953-787a-49a2-87d5-1514f1c425ee-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:26.645505 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.645511 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d29953-787a-49a2-87d5-1514f1c425ee-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:26.645714 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:26.645520 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d29953-787a-49a2-87d5-1514f1c425ee-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:27.158938 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:27.158906 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:27.264393 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:27.264352 5300 generic.go:356] "Generic (PLEG): container finished" podID="85d29953-787a-49a2-87d5-1514f1c425ee" containerID="a12c568c8faf5863370778a422fd81d0ad39bbb2cad6c9c404a8b0fdf8a94bcc" exitCode=0 Oct 30 01:30:27.264599 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:27.264421 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" Oct 30 01:30:27.264599 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:27.264483 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" event={"ID":"85d29953-787a-49a2-87d5-1514f1c425ee","Type":"ContainerDied","Data":"a12c568c8faf5863370778a422fd81d0ad39bbb2cad6c9c404a8b0fdf8a94bcc"} Oct 30 01:30:27.264599 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:27.264527 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d" event={"ID":"85d29953-787a-49a2-87d5-1514f1c425ee","Type":"ContainerDied","Data":"0e828edd81cefabe499dcccd4382d5bad6c6b48f62e4fbefd5f1b4d7ce17ef79"} Oct 30 01:30:27.264599 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:27.264555 5300 scope.go:122] "RemoveContainer" containerID="a12c568c8faf5863370778a422fd81d0ad39bbb2cad6c9c404a8b0fdf8a94bcc" Oct 30 01:30:27.278385 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:27.278363 5300 scope.go:122] "RemoveContainer" containerID="941f4f0b7c5f816640828673b59e1bad24d087dfbfe2dc5a407cec115d803b8b" Oct 30 01:30:27.288353 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:27.288333 5300 scope.go:122] "RemoveContainer" containerID="a12c568c8faf5863370778a422fd81d0ad39bbb2cad6c9c404a8b0fdf8a94bcc" Oct 30 01:30:27.288697 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:30:27.288667 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a12c568c8faf5863370778a422fd81d0ad39bbb2cad6c9c404a8b0fdf8a94bcc\": container with ID starting with a12c568c8faf5863370778a422fd81d0ad39bbb2cad6c9c404a8b0fdf8a94bcc not found: ID does not exist" containerID="a12c568c8faf5863370778a422fd81d0ad39bbb2cad6c9c404a8b0fdf8a94bcc" Oct 30 01:30:27.288850 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:27.288708 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a12c568c8faf5863370778a422fd81d0ad39bbb2cad6c9c404a8b0fdf8a94bcc"} err="failed to get container status \"a12c568c8faf5863370778a422fd81d0ad39bbb2cad6c9c404a8b0fdf8a94bcc\": rpc error: code = NotFound desc = could not find container \"a12c568c8faf5863370778a422fd81d0ad39bbb2cad6c9c404a8b0fdf8a94bcc\": container with ID starting with a12c568c8faf5863370778a422fd81d0ad39bbb2cad6c9c404a8b0fdf8a94bcc not found: ID does not exist" Oct 30 01:30:27.288850 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:27.288732 5300 scope.go:122] "RemoveContainer" containerID="941f4f0b7c5f816640828673b59e1bad24d087dfbfe2dc5a407cec115d803b8b" Oct 30 01:30:27.289189 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:30:27.289170 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"941f4f0b7c5f816640828673b59e1bad24d087dfbfe2dc5a407cec115d803b8b\": container with ID starting with 941f4f0b7c5f816640828673b59e1bad24d087dfbfe2dc5a407cec115d803b8b not found: ID does not exist" containerID="941f4f0b7c5f816640828673b59e1bad24d087dfbfe2dc5a407cec115d803b8b" Oct 30 01:30:27.289274 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:27.289193 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"941f4f0b7c5f816640828673b59e1bad24d087dfbfe2dc5a407cec115d803b8b"} err="failed to get container status \"941f4f0b7c5f816640828673b59e1bad24d087dfbfe2dc5a407cec115d803b8b\": rpc error: code = NotFound desc = could not find container \"941f4f0b7c5f816640828673b59e1bad24d087dfbfe2dc5a407cec115d803b8b\": container with ID starting with 941f4f0b7c5f816640828673b59e1bad24d087dfbfe2dc5a407cec115d803b8b not found: ID does not exist" Oct 30 01:30:27.338232 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:27.338199 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d"] Oct 30 01:30:27.341793 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:27.341751 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:27.352138 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:27.352113 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-368bbb-openshift-default-7bb67896f-mq44d"] Oct 30 01:30:27.524915 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:27.524878 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:27.821452 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:27.821371 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85d29953-787a-49a2-87d5-1514f1c425ee" path="/var/lib/kubelet/pods/85d29953-787a-49a2-87d5-1514f1c425ee/volumes" Oct 30 01:30:27.834424 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:27.834391 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb" podUID="7f309af2-1f23-4295-8187-129275259d8a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:30:27.834544 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:27.834396 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm" podUID="5c19f786-50e8-4dee-97ae-c4445035f930" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:30:28.027336 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:28.027303 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:28.383077 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:28.383036 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:28.445458 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:28.445421 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerName="main" containerID="cri-o://ba2aa076b6b05abc96d46bacb29fc7ea1d4d0c199e211046a9de96b50c20583d" gracePeriod=45 Oct 30 01:30:28.730926 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:28.730900 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 01:30:28.787866 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:28.787832 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d4a7520-7709-4747-b6b1-dda7c86290c1-kube-api-access-g28nl\" (UniqueName: \"kubernetes.io/projected/4d4a7520-7709-4747-b6b1-dda7c86290c1-kube-api-access-g28nl\") pod \"4d4a7520-7709-4747-b6b1-dda7c86290c1\" (UID: \"4d4a7520-7709-4747-b6b1-dda7c86290c1\") " Oct 30 01:30:28.788033 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:28.787952 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-dshm\") pod \"4d4a7520-7709-4747-b6b1-dda7c86290c1\" (UID: \"4d4a7520-7709-4747-b6b1-dda7c86290c1\") " Oct 30 01:30:28.788119 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:28.788038 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d4a7520-7709-4747-b6b1-dda7c86290c1-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4a7520-7709-4747-b6b1-dda7c86290c1-tls-certs\") pod \"4d4a7520-7709-4747-b6b1-dda7c86290c1\" (UID: \"4d4a7520-7709-4747-b6b1-dda7c86290c1\") " Oct 30 01:30:28.788119 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:28.788057 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-tmp-dir\") pod \"4d4a7520-7709-4747-b6b1-dda7c86290c1\" (UID: \"4d4a7520-7709-4747-b6b1-dda7c86290c1\") " Oct 30 01:30:28.788119 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:28.788081 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-home\" (UniqueName: \"kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-home\") pod \"4d4a7520-7709-4747-b6b1-dda7c86290c1\" (UID: \"4d4a7520-7709-4747-b6b1-dda7c86290c1\") " Oct 30 01:30:28.788290 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:28.788140 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-model-cache\") pod \"4d4a7520-7709-4747-b6b1-dda7c86290c1\" (UID: \"4d4a7520-7709-4747-b6b1-dda7c86290c1\") " Oct 30 01:30:28.788510 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:28.788481 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-tmp-dir" pod "4d4a7520-7709-4747-b6b1-dda7c86290c1" (UID: "4d4a7520-7709-4747-b6b1-dda7c86290c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:28.788700 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:28.788641 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-home" pod "4d4a7520-7709-4747-b6b1-dda7c86290c1" (UID: "4d4a7520-7709-4747-b6b1-dda7c86290c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:28.788700 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:28.788656 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-model-cache" pod "4d4a7520-7709-4747-b6b1-dda7c86290c1" (UID: "4d4a7520-7709-4747-b6b1-dda7c86290c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:28.788909 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:28.788879 5300 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-tmp-dir\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:28.788909 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:28.788900 5300 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-home\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:28.788984 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:28.788914 5300 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-model-cache\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:28.790442 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:28.790404 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d4a7520-7709-4747-b6b1-dda7c86290c1-tls-certs" pod "4d4a7520-7709-4747-b6b1-dda7c86290c1" (UID: "4d4a7520-7709-4747-b6b1-dda7c86290c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:28.790553 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:28.790515 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d4a7520-7709-4747-b6b1-dda7c86290c1-kube-api-access-g28nl" pod "4d4a7520-7709-4747-b6b1-dda7c86290c1" (UID: "4d4a7520-7709-4747-b6b1-dda7c86290c1"). InnerVolumeSpecName "kube-api-access-g28nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:28.790906 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:28.790877 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-dshm" pod "4d4a7520-7709-4747-b6b1-dda7c86290c1" (UID: "4d4a7520-7709-4747-b6b1-dda7c86290c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:28.890883 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:28.890851 5300 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4a7520-7709-4747-b6b1-dda7c86290c1-dshm\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:28.890883 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:28.890882 5300 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4a7520-7709-4747-b6b1-dda7c86290c1-tls-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:28.891071 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:28.890894 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g28nl\" (UniqueName: \"kubernetes.io/projected/4d4a7520-7709-4747-b6b1-dda7c86290c1-kube-api-access-g28nl\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:29.076595 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.076559 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq"] Oct 30 01:30:29.076861 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.076833 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq" podUID="c2839a72-aea8-412d-b105-cf819e2e497a" containerName="maas-api" containerID="cri-o://db74a89c8147cee04cb6ae268efdb792ab6936ad1a2051ba37f0d761bcf4570d" gracePeriod=30 Oct 30 01:30:29.194133 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.194103 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:29.285381 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.285342 5300 generic.go:356] "Generic (PLEG): container finished" podID="c2839a72-aea8-412d-b105-cf819e2e497a" containerID="db74a89c8147cee04cb6ae268efdb792ab6936ad1a2051ba37f0d761bcf4570d" exitCode=0 Oct 30 01:30:29.285554 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.285419 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq" event={"ID":"c2839a72-aea8-412d-b105-cf819e2e497a","Type":"ContainerDied","Data":"db74a89c8147cee04cb6ae268efdb792ab6936ad1a2051ba37f0d761bcf4570d"} Oct 30 01:30:29.287236 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.287207 5300 generic.go:356] "Generic (PLEG): container finished" podID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerID="ba2aa076b6b05abc96d46bacb29fc7ea1d4d0c199e211046a9de96b50c20583d" exitCode=0 Oct 30 01:30:29.287382 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.287265 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" Oct 30 01:30:29.287382 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.287277 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" event={"ID":"4d4a7520-7709-4747-b6b1-dda7c86290c1","Type":"ContainerDied","Data":"ba2aa076b6b05abc96d46bacb29fc7ea1d4d0c199e211046a9de96b50c20583d"} Oct 30 01:30:29.287382 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.287320 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q" event={"ID":"4d4a7520-7709-4747-b6b1-dda7c86290c1","Type":"ContainerDied","Data":"45b234972420fd72874f4dd6a99db68269aa16ddfbebe72f330f56b7dae72a83"} Oct 30 01:30:29.287382 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.287342 5300 scope.go:122] "RemoveContainer" containerID="ba2aa076b6b05abc96d46bacb29fc7ea1d4d0c199e211046a9de96b50c20583d" Oct 30 01:30:29.299134 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.299021 5300 scope.go:122] "RemoveContainer" containerID="ba2aa076b6b05abc96d46bacb29fc7ea1d4d0c199e211046a9de96b50c20583d" Oct 30 01:30:29.299342 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:30:29.299321 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba2aa076b6b05abc96d46bacb29fc7ea1d4d0c199e211046a9de96b50c20583d\": container with ID starting with ba2aa076b6b05abc96d46bacb29fc7ea1d4d0c199e211046a9de96b50c20583d not found: ID does not exist" containerID="ba2aa076b6b05abc96d46bacb29fc7ea1d4d0c199e211046a9de96b50c20583d" Oct 30 01:30:29.299421 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.299354 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba2aa076b6b05abc96d46bacb29fc7ea1d4d0c199e211046a9de96b50c20583d"} err="failed to get container status \"ba2aa076b6b05abc96d46bacb29fc7ea1d4d0c199e211046a9de96b50c20583d\": rpc error: code = NotFound desc = could not find container \"ba2aa076b6b05abc96d46bacb29fc7ea1d4d0c199e211046a9de96b50c20583d\": container with ID starting with ba2aa076b6b05abc96d46bacb29fc7ea1d4d0c199e211046a9de96b50c20583d not found: ID does not exist" Oct 30 01:30:29.396152 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.378435 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q"] Oct 30 01:30:29.396152 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.378494 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-dca38d/premium-w5-dca38d-kserve-6c579c47c4-mqn6q"] Oct 30 01:30:29.398046 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.398017 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq" Oct 30 01:30:29.498629 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.498594 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2839a72-aea8-412d-b105-cf819e2e497a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2839a72-aea8-412d-b105-cf819e2e497a-openshift-service-ca\") pod \"c2839a72-aea8-412d-b105-cf819e2e497a\" (UID: \"c2839a72-aea8-412d-b105-cf819e2e497a\") " Oct 30 01:30:29.498938 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.498639 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2839a72-aea8-412d-b105-cf819e2e497a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2839a72-aea8-412d-b105-cf819e2e497a-metrics-certs\") pod \"c2839a72-aea8-412d-b105-cf819e2e497a\" (UID: \"c2839a72-aea8-412d-b105-cf819e2e497a\") " Oct 30 01:30:29.498938 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.498842 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2839a72-aea8-412d-b105-cf819e2e497a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2839a72-aea8-412d-b105-cf819e2e497a-maas-api-tls\") pod \"c2839a72-aea8-412d-b105-cf819e2e497a\" (UID: \"c2839a72-aea8-412d-b105-cf819e2e497a\") " Oct 30 01:30:29.498938 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.498924 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2839a72-aea8-412d-b105-cf819e2e497a-kube-api-access-mvwrb\" (UniqueName: \"kubernetes.io/projected/c2839a72-aea8-412d-b105-cf819e2e497a-kube-api-access-mvwrb\") pod \"c2839a72-aea8-412d-b105-cf819e2e497a\" (UID: \"c2839a72-aea8-412d-b105-cf819e2e497a\") " Oct 30 01:30:29.499150 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.499087 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2839a72-aea8-412d-b105-cf819e2e497a-openshift-service-ca" pod "c2839a72-aea8-412d-b105-cf819e2e497a" (UID: "c2839a72-aea8-412d-b105-cf819e2e497a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:29.499491 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.499463 5300 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2839a72-aea8-412d-b105-cf819e2e497a-openshift-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:29.501197 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.501163 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2839a72-aea8-412d-b105-cf819e2e497a-kube-api-access-mvwrb" pod "c2839a72-aea8-412d-b105-cf819e2e497a" (UID: "c2839a72-aea8-412d-b105-cf819e2e497a"). InnerVolumeSpecName "kube-api-access-mvwrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:29.501310 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.501206 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2839a72-aea8-412d-b105-cf819e2e497a-maas-api-tls" pod "c2839a72-aea8-412d-b105-cf819e2e497a" (UID: "c2839a72-aea8-412d-b105-cf819e2e497a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:29.501744 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.501720 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2839a72-aea8-412d-b105-cf819e2e497a-metrics-certs" pod "c2839a72-aea8-412d-b105-cf819e2e497a" (UID: "c2839a72-aea8-412d-b105-cf819e2e497a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:29.600512 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.600428 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2839a72-aea8-412d-b105-cf819e2e497a-maas-api-tls\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:29.600512 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.600459 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvwrb\" (UniqueName: \"kubernetes.io/projected/c2839a72-aea8-412d-b105-cf819e2e497a-kube-api-access-mvwrb\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:29.600512 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.600469 5300 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2839a72-aea8-412d-b105-cf819e2e497a-metrics-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:29.775953 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.775919 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:29.821241 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.821210 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" path="/var/lib/kubelet/pods/4d4a7520-7709-4747-b6b1-dda7c86290c1/volumes" Oct 30 01:30:29.968294 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:29.968209 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:30.296272 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:30.296236 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq" event={"ID":"c2839a72-aea8-412d-b105-cf819e2e497a","Type":"ContainerDied","Data":"8b9ec5e6f6c1e12058058b4b0b46bfc2f350d7658a40068d00c137c71d4b12f9"} Oct 30 01:30:30.296464 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:30.296287 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq" Oct 30 01:30:30.296464 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:30.296321 5300 scope.go:122] "RemoveContainer" containerID="db74a89c8147cee04cb6ae268efdb792ab6936ad1a2051ba37f0d761bcf4570d" Oct 30 01:30:30.366805 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:30.366770 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq"] Oct 30 01:30:30.381051 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:30.381021 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d827b7-69848b5f7b-xhmxq"] Oct 30 01:30:30.453741 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:30.453704 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:31.712869 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:31.712834 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz"] Oct 30 01:30:31.713274 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:31.713105 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz" podUID="ec2520e7-a5f7-4127-ac4b-5bb40efc6309" containerName="maas-api" containerID="cri-o://4a4e04ea7a262a8ee834fbce9222ef83088803fc9313d6e12fdd441f7fc36d24" gracePeriod=30 Oct 30 01:30:31.777380 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:31.777345 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:31.821573 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:31.821543 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2839a72-aea8-412d-b105-cf819e2e497a" path="/var/lib/kubelet/pods/c2839a72-aea8-412d-b105-cf819e2e497a/volumes" Oct 30 01:30:32.006914 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:32.006889 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz" Oct 30 01:30:32.038220 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:32.038186 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-maas-api-tls\") pod \"ec2520e7-a5f7-4127-ac4b-5bb40efc6309\" (UID: \"ec2520e7-a5f7-4127-ac4b-5bb40efc6309\") " Oct 30 01:30:32.038390 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:32.038376 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-openshift-service-ca\") pod \"ec2520e7-a5f7-4127-ac4b-5bb40efc6309\" (UID: \"ec2520e7-a5f7-4127-ac4b-5bb40efc6309\") " Oct 30 01:30:32.038473 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:32.038419 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-kube-api-access-2v2jk\" (UniqueName: \"kubernetes.io/projected/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-kube-api-access-2v2jk\") pod \"ec2520e7-a5f7-4127-ac4b-5bb40efc6309\" (UID: \"ec2520e7-a5f7-4127-ac4b-5bb40efc6309\") " Oct 30 01:30:32.038473 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:32.038445 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-metrics-certs\") pod \"ec2520e7-a5f7-4127-ac4b-5bb40efc6309\" (UID: \"ec2520e7-a5f7-4127-ac4b-5bb40efc6309\") " Oct 30 01:30:32.038838 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:32.038801 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-openshift-service-ca" pod "ec2520e7-a5f7-4127-ac4b-5bb40efc6309" (UID: "ec2520e7-a5f7-4127-ac4b-5bb40efc6309"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:32.038976 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:32.038937 5300 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-openshift-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:32.040430 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:32.040404 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-maas-api-tls" pod "ec2520e7-a5f7-4127-ac4b-5bb40efc6309" (UID: "ec2520e7-a5f7-4127-ac4b-5bb40efc6309"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:32.040554 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:32.040538 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-metrics-certs" pod "ec2520e7-a5f7-4127-ac4b-5bb40efc6309" (UID: "ec2520e7-a5f7-4127-ac4b-5bb40efc6309"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:32.040554 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:32.040541 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-kube-api-access-2v2jk" pod "ec2520e7-a5f7-4127-ac4b-5bb40efc6309" (UID: "ec2520e7-a5f7-4127-ac4b-5bb40efc6309"). InnerVolumeSpecName "kube-api-access-2v2jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:32.048393 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:32.048369 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:32.141700 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:32.141259 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v2jk\" (UniqueName: \"kubernetes.io/projected/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-kube-api-access-2v2jk\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:32.141700 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:32.141296 5300 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-metrics-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:32.141700 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:32.141309 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2520e7-a5f7-4127-ac4b-5bb40efc6309-maas-api-tls\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:32.366182 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:32.346264 5300 generic.go:356] "Generic (PLEG): container finished" podID="ec2520e7-a5f7-4127-ac4b-5bb40efc6309" containerID="4a4e04ea7a262a8ee834fbce9222ef83088803fc9313d6e12fdd441f7fc36d24" exitCode=0 Oct 30 01:30:32.366182 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:32.346477 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz" event={"ID":"ec2520e7-a5f7-4127-ac4b-5bb40efc6309","Type":"ContainerDied","Data":"4a4e04ea7a262a8ee834fbce9222ef83088803fc9313d6e12fdd441f7fc36d24"} Oct 30 01:30:32.366182 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:32.346526 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz" event={"ID":"ec2520e7-a5f7-4127-ac4b-5bb40efc6309","Type":"ContainerDied","Data":"78db8494920cb95221a6932de507bab57822a21b6172fcb6bf8fd28cd14cfc67"} Oct 30 01:30:32.366182 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:32.346556 5300 scope.go:122] "RemoveContainer" containerID="4a4e04ea7a262a8ee834fbce9222ef83088803fc9313d6e12fdd441f7fc36d24" Oct 30 01:30:32.366182 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:32.346682 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz" Oct 30 01:30:32.379389 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:32.379354 5300 scope.go:122] "RemoveContainer" containerID="4a4e04ea7a262a8ee834fbce9222ef83088803fc9313d6e12fdd441f7fc36d24" Oct 30 01:30:32.382232 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:30:32.382194 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a4e04ea7a262a8ee834fbce9222ef83088803fc9313d6e12fdd441f7fc36d24\": container with ID starting with 4a4e04ea7a262a8ee834fbce9222ef83088803fc9313d6e12fdd441f7fc36d24 not found: ID does not exist" containerID="4a4e04ea7a262a8ee834fbce9222ef83088803fc9313d6e12fdd441f7fc36d24" Oct 30 01:30:32.386466 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:32.386428 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4e04ea7a262a8ee834fbce9222ef83088803fc9313d6e12fdd441f7fc36d24"} err="failed to get container status \"4a4e04ea7a262a8ee834fbce9222ef83088803fc9313d6e12fdd441f7fc36d24\": rpc error: code = NotFound desc = could not find container \"4a4e04ea7a262a8ee834fbce9222ef83088803fc9313d6e12fdd441f7fc36d24\": container with ID starting with 4a4e04ea7a262a8ee834fbce9222ef83088803fc9313d6e12fdd441f7fc36d24 not found: ID does not exist" Oct 30 01:30:32.392995 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:32.392970 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:32.438211 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:32.438176 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz"] Oct 30 01:30:32.454114 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:32.454084 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c104b8-74685b8747-cbkdz"] Oct 30 01:30:32.622628 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:32.622546 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:32.839546 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:32.839515 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:33.226358 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:33.226331 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:33.388640 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:33.388607 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:33.823043 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:33.823012 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec2520e7-a5f7-4127-ac4b-5bb40efc6309" path="/var/lib/kubelet/pods/ec2520e7-a5f7-4127-ac4b-5bb40efc6309/volumes" Oct 30 01:30:34.541579 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:34.541544 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:34.821380 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:34.821289 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:34.939694 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:34.939664 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj"] Oct 30 01:30:34.939892 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:34.939872 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" podUID="ba5bee00-9a0a-4a43-abea-35da290cbcaa" containerName="istio-proxy" containerID="cri-o://f4f1bbe61a07b1e4d1552f5c783cc5daef455ccc9eae727d360391c0b1c64fb9" gracePeriod=30 Oct 30 01:30:35.074334 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:35.069726 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:35.586981 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:35.586950 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:36.888947 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:36.888913 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:37.834568 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:37.834526 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm" podUID="5c19f786-50e8-4dee-97ae-c4445035f930" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:30:37.834568 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:37.834548 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb" podUID="7f309af2-1f23-4295-8187-129275259d8a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:30:38.600897 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:38.600842 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:38.877717 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:38.877619 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:39.118202 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:39.118169 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:39.231277 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:39.231250 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:39.372468 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:39.372429 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:39.897754 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:39.897723 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:40.058261 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.058227 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:40.224825 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.224796 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 01:30:40.259748 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.259717 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba5bee00-9a0a-4a43-abea-35da290cbcaa-kube-api-access-r6zjq\" (UniqueName: \"kubernetes.io/projected/ba5bee00-9a0a-4a43-abea-35da290cbcaa-kube-api-access-r6zjq\") pod \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " Oct 30 01:30:40.259980 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.259818 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-data\") pod \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " Oct 30 01:30:40.259980 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.259848 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-envoy\") pod \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " Oct 30 01:30:40.259980 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.259889 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-workload-socket\") pod \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " Oct 30 01:30:40.260155 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.260009 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-credential-socket\") pod \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " Oct 30 01:30:40.260155 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.260118 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-podinfo\") pod \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " Oct 30 01:30:40.260273 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.260158 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-data" pod "ba5bee00-9a0a-4a43-abea-35da290cbcaa" (UID: "ba5bee00-9a0a-4a43-abea-35da290cbcaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:40.260273 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.260196 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-token\" (UniqueName: \"kubernetes.io/projected/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-token\") pod \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " Oct 30 01:30:40.260273 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.260225 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-workload-socket" pod "ba5bee00-9a0a-4a43-abea-35da290cbcaa" (UID: "ba5bee00-9a0a-4a43-abea-35da290cbcaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:40.260273 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.260250 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istiod-ca-cert\") pod \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " Oct 30 01:30:40.260273 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.260256 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-credential-socket" pod "ba5bee00-9a0a-4a43-abea-35da290cbcaa" (UID: "ba5bee00-9a0a-4a43-abea-35da290cbcaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:40.260585 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.260301 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-workload-certs\") pod \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\" (UID: \"ba5bee00-9a0a-4a43-abea-35da290cbcaa\") " Oct 30 01:30:40.260671 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.260636 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-workload-certs" pod "ba5bee00-9a0a-4a43-abea-35da290cbcaa" (UID: "ba5bee00-9a0a-4a43-abea-35da290cbcaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:40.260732 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.260666 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istiod-ca-cert" pod "ba5bee00-9a0a-4a43-abea-35da290cbcaa" (UID: "ba5bee00-9a0a-4a43-abea-35da290cbcaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:40.261016 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.260993 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:40.261082 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.261033 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:40.261082 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.261048 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:40.261082 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.261062 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:40.261082 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.261078 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:40.262343 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.262321 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-podinfo" pod "ba5bee00-9a0a-4a43-abea-35da290cbcaa" (UID: "ba5bee00-9a0a-4a43-abea-35da290cbcaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:30:40.262465 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.262433 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-token" pod "ba5bee00-9a0a-4a43-abea-35da290cbcaa" (UID: "ba5bee00-9a0a-4a43-abea-35da290cbcaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:40.262518 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.262490 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-envoy" pod "ba5bee00-9a0a-4a43-abea-35da290cbcaa" (UID: "ba5bee00-9a0a-4a43-abea-35da290cbcaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:40.262553 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.262523 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5bee00-9a0a-4a43-abea-35da290cbcaa-kube-api-access-r6zjq" pod "ba5bee00-9a0a-4a43-abea-35da290cbcaa" (UID: "ba5bee00-9a0a-4a43-abea-35da290cbcaa"). InnerVolumeSpecName "kube-api-access-r6zjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:40.282097 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.282063 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:40.362976 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.362940 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:40.362976 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.362966 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:40.362976 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.362979 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6zjq\" (UniqueName: \"kubernetes.io/projected/ba5bee00-9a0a-4a43-abea-35da290cbcaa-kube-api-access-r6zjq\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:40.363207 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.362988 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5bee00-9a0a-4a43-abea-35da290cbcaa-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:40.426406 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.426370 5300 generic.go:356] "Generic (PLEG): container finished" podID="ba5bee00-9a0a-4a43-abea-35da290cbcaa" containerID="f4f1bbe61a07b1e4d1552f5c783cc5daef455ccc9eae727d360391c0b1c64fb9" exitCode=0 Oct 30 01:30:40.426610 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.426451 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" Oct 30 01:30:40.426610 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.426449 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" event={"ID":"ba5bee00-9a0a-4a43-abea-35da290cbcaa","Type":"ContainerDied","Data":"f4f1bbe61a07b1e4d1552f5c783cc5daef455ccc9eae727d360391c0b1c64fb9"} Oct 30 01:30:40.426741 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.426624 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj" event={"ID":"ba5bee00-9a0a-4a43-abea-35da290cbcaa","Type":"ContainerDied","Data":"0ceaec7a059d78008637a5d7948568fee9fb38e31adcaaececa6b3e0332ac902"} Oct 30 01:30:40.426741 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.426660 5300 scope.go:122] "RemoveContainer" containerID="f4f1bbe61a07b1e4d1552f5c783cc5daef455ccc9eae727d360391c0b1c64fb9" Oct 30 01:30:40.440489 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.440464 5300 scope.go:122] "RemoveContainer" containerID="f4f1bbe61a07b1e4d1552f5c783cc5daef455ccc9eae727d360391c0b1c64fb9" Oct 30 01:30:40.440876 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:30:40.440839 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4f1bbe61a07b1e4d1552f5c783cc5daef455ccc9eae727d360391c0b1c64fb9\": container with ID starting with f4f1bbe61a07b1e4d1552f5c783cc5daef455ccc9eae727d360391c0b1c64fb9 not found: ID does not exist" containerID="f4f1bbe61a07b1e4d1552f5c783cc5daef455ccc9eae727d360391c0b1c64fb9" Oct 30 01:30:40.440982 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.440882 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4f1bbe61a07b1e4d1552f5c783cc5daef455ccc9eae727d360391c0b1c64fb9"} err="failed to get container status \"f4f1bbe61a07b1e4d1552f5c783cc5daef455ccc9eae727d360391c0b1c64fb9\": rpc error: code = NotFound desc = could not find container \"f4f1bbe61a07b1e4d1552f5c783cc5daef455ccc9eae727d360391c0b1c64fb9\": container with ID starting with f4f1bbe61a07b1e4d1552f5c783cc5daef455ccc9eae727d360391c0b1c64fb9 not found: ID does not exist" Oct 30 01:30:40.493283 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.493253 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj"] Oct 30 01:30:40.500967 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.500939 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fd3281-openshift-default-75dc5c9669-6n6tj"] Oct 30 01:30:40.765948 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:40.765913 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:41.821481 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:41.821449 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba5bee00-9a0a-4a43-abea-35da290cbcaa" path="/var/lib/kubelet/pods/ba5bee00-9a0a-4a43-abea-35da290cbcaa/volumes" Oct 30 01:30:41.841003 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:41.840973 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:42.055216 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:42.055175 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:42.204991 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:42.204902 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8"] Oct 30 01:30:42.205184 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:42.205157 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" podUID="db62d92e-f4b6-480b-9545-569baf9670a8" containerName="istio-proxy" containerID="cri-o://e3f0420d312b02f08636fe9caf4ea8fb2200987eba800ca086ff703ebc5a6fa3" gracePeriod=30 Oct 30 01:30:42.293934 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:42.293900 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:42.486599 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:42.486561 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:42.636140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:42.636103 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:43.077545 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:43.077483 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:43.741582 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:43.741550 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:44.225166 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.225120 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:44.253938 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.253912 5300 patch_prober.go:29] interesting pod/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:30:44.254058 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.253971 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" podUID="db62d92e-f4b6-480b-9545-569baf9670a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:30:44.453855 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.453473 5300 generic.go:356] "Generic (PLEG): container finished" podID="7f309af2-1f23-4295-8187-129275259d8a" containerID="981208c4e05bf33db5ad768118b71b789f8794874cea061663aa65a16be50af5" exitCode=0 Oct 30 01:30:44.453855 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.453577 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb" event={"ID":"7f309af2-1f23-4295-8187-129275259d8a","Type":"ContainerDied","Data":"981208c4e05bf33db5ad768118b71b789f8794874cea061663aa65a16be50af5"} Oct 30 01:30:44.453855 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.453631 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb" event={"ID":"7f309af2-1f23-4295-8187-129275259d8a","Type":"ContainerDied","Data":"8af764958954d12ee568c25be441a0ed8952eea3ffe786eb8e37403179ba69c6"} Oct 30 01:30:44.453855 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.453657 5300 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8af764958954d12ee568c25be441a0ed8952eea3ffe786eb8e37403179ba69c6" Oct 30 01:30:44.456430 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.456186 5300 generic.go:356] "Generic (PLEG): container finished" podID="5c19f786-50e8-4dee-97ae-c4445035f930" containerID="afbf1873ed4296546c649b91a42b600c578726cb07be0da8c37020648290bcdb" exitCode=0 Oct 30 01:30:44.456430 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.456267 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm" event={"ID":"5c19f786-50e8-4dee-97ae-c4445035f930","Type":"ContainerDied","Data":"afbf1873ed4296546c649b91a42b600c578726cb07be0da8c37020648290bcdb"} Oct 30 01:30:44.476445 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.476418 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb" Oct 30 01:30:44.490627 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.490590 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:44.512988 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.512964 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f309af2-1f23-4295-8187-129275259d8a-kube-api-access-bcwvs\" (UniqueName: \"kubernetes.io/projected/7f309af2-1f23-4295-8187-129275259d8a-kube-api-access-bcwvs\") pod \"7f309af2-1f23-4295-8187-129275259d8a\" (UID: \"7f309af2-1f23-4295-8187-129275259d8a\") " Oct 30 01:30:44.513107 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.513084 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f309af2-1f23-4295-8187-129275259d8a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f309af2-1f23-4295-8187-129275259d8a-plugins-config-volume\") pod \"7f309af2-1f23-4295-8187-129275259d8a\" (UID: \"7f309af2-1f23-4295-8187-129275259d8a\") " Oct 30 01:30:44.513469 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.513447 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f309af2-1f23-4295-8187-129275259d8a-plugins-config-volume" pod "7f309af2-1f23-4295-8187-129275259d8a" (UID: "7f309af2-1f23-4295-8187-129275259d8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:44.515195 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.515175 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f309af2-1f23-4295-8187-129275259d8a-kube-api-access-bcwvs" pod "7f309af2-1f23-4295-8187-129275259d8a" (UID: "7f309af2-1f23-4295-8187-129275259d8a"). InnerVolumeSpecName "kube-api-access-bcwvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:44.528294 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.528271 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm" Oct 30 01:30:44.614137 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.614104 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c19f786-50e8-4dee-97ae-c4445035f930-kube-api-access-p5gbc\" (UniqueName: \"kubernetes.io/projected/5c19f786-50e8-4dee-97ae-c4445035f930-kube-api-access-p5gbc\") pod \"5c19f786-50e8-4dee-97ae-c4445035f930\" (UID: \"5c19f786-50e8-4dee-97ae-c4445035f930\") " Oct 30 01:30:44.614327 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.614205 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c19f786-50e8-4dee-97ae-c4445035f930-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c19f786-50e8-4dee-97ae-c4445035f930-plugins-config-volume\") pod \"5c19f786-50e8-4dee-97ae-c4445035f930\" (UID: \"5c19f786-50e8-4dee-97ae-c4445035f930\") " Oct 30 01:30:44.614694 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.614659 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c19f786-50e8-4dee-97ae-c4445035f930-plugins-config-volume" pod "5c19f786-50e8-4dee-97ae-c4445035f930" (UID: "5c19f786-50e8-4dee-97ae-c4445035f930"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:44.614933 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.614908 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c19f786-50e8-4dee-97ae-c4445035f930-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:44.615031 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.614938 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcwvs\" (UniqueName: \"kubernetes.io/projected/7f309af2-1f23-4295-8187-129275259d8a-kube-api-access-bcwvs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:44.615031 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.614953 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f309af2-1f23-4295-8187-129275259d8a-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:44.616486 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.616457 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c19f786-50e8-4dee-97ae-c4445035f930-kube-api-access-p5gbc" pod "5c19f786-50e8-4dee-97ae-c4445035f930" (UID: "5c19f786-50e8-4dee-97ae-c4445035f930"). InnerVolumeSpecName "kube-api-access-p5gbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:44.682188 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.682158 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:44.716515 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.716442 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5gbc\" (UniqueName: \"kubernetes.io/projected/5c19f786-50e8-4dee-97ae-c4445035f930-kube-api-access-p5gbc\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:44.915573 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.914262 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:44.980451 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.980413 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv"] Oct 30 01:30:44.980733 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:44.980703 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" podUID="78eeb03d-9184-4a3d-af85-b8391731e828" containerName="istio-proxy" containerID="cri-o://2b4e8c4555bb286dae11f98457625a1e56a08fba51673d1c66aecdc08d31cbf8" gracePeriod=30 Oct 30 01:30:45.081115 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:45.081082 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:45.463987 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:45.463894 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm" Oct 30 01:30:45.463987 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:45.463916 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm" event={"ID":"5c19f786-50e8-4dee-97ae-c4445035f930","Type":"ContainerDied","Data":"8827f2bf0bd025094fb6543d301a8b35694a7e97163dc3ab93625c9d24620694"} Oct 30 01:30:45.464535 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:45.463988 5300 scope.go:122] "RemoveContainer" containerID="afbf1873ed4296546c649b91a42b600c578726cb07be0da8c37020648290bcdb" Oct 30 01:30:45.464535 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:45.464078 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb" Oct 30 01:30:45.555468 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:45.555435 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb"] Oct 30 01:30:45.569413 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:45.569385 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-dca38d-745fb76666-xb8fb"] Oct 30 01:30:45.607356 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:45.607324 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm"] Oct 30 01:30:45.609652 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:45.609632 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-dca38d-6f88b7b555-kknsm"] Oct 30 01:30:45.820079 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:45.820046 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c19f786-50e8-4dee-97ae-c4445035f930" path="/var/lib/kubelet/pods/5c19f786-50e8-4dee-97ae-c4445035f930/volumes" Oct 30 01:30:45.820594 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:45.820573 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f309af2-1f23-4295-8187-129275259d8a" path="/var/lib/kubelet/pods/7f309af2-1f23-4295-8187-129275259d8a/volumes" Oct 30 01:30:47.423321 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.411392 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:47.480615 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.480589 5300 generic.go:356] "Generic (PLEG): container finished" podID="db62d92e-f4b6-480b-9545-569baf9670a8" containerID="e3f0420d312b02f08636fe9caf4ea8fb2200987eba800ca086ff703ebc5a6fa3" exitCode=0 Oct 30 01:30:47.480756 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.480652 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" event={"ID":"db62d92e-f4b6-480b-9545-569baf9670a8","Type":"ContainerDied","Data":"e3f0420d312b02f08636fe9caf4ea8fb2200987eba800ca086ff703ebc5a6fa3"} Oct 30 01:30:47.480756 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.480709 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" event={"ID":"db62d92e-f4b6-480b-9545-569baf9670a8","Type":"ContainerDied","Data":"f95f96b2a6f30bbf671492b34139903272ecc8a206c382a3572d73069614fd99"} Oct 30 01:30:47.480756 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.480726 5300 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f95f96b2a6f30bbf671492b34139903272ecc8a206c382a3572d73069614fd99" Oct 30 01:30:47.497145 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.497124 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 01:30:47.558859 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.558824 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-istio-data\") pod \"db62d92e-f4b6-480b-9545-569baf9670a8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " Oct 30 01:30:47.559060 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.558957 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db62d92e-f4b6-480b-9545-569baf9670a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db62d92e-f4b6-480b-9545-569baf9670a8-istio-podinfo\") pod \"db62d92e-f4b6-480b-9545-569baf9670a8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " Oct 30 01:30:47.559060 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.559011 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-workload-certs\") pod \"db62d92e-f4b6-480b-9545-569baf9670a8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " Oct 30 01:30:47.559060 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.559050 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db62d92e-f4b6-480b-9545-569baf9670a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db62d92e-f4b6-480b-9545-569baf9670a8-istiod-ca-cert\") pod \"db62d92e-f4b6-480b-9545-569baf9670a8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " Oct 30 01:30:47.559238 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.559086 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-istio-data" pod "db62d92e-f4b6-480b-9545-569baf9670a8" (UID: "db62d92e-f4b6-480b-9545-569baf9670a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:47.559238 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.559147 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db62d92e-f4b6-480b-9545-569baf9670a8-kube-api-access-km6w9\" (UniqueName: \"kubernetes.io/projected/db62d92e-f4b6-480b-9545-569baf9670a8-kube-api-access-km6w9\") pod \"db62d92e-f4b6-480b-9545-569baf9670a8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " Oct 30 01:30:47.559238 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.559180 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-istio-envoy\") pod \"db62d92e-f4b6-480b-9545-569baf9670a8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " Oct 30 01:30:47.559404 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.559241 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db62d92e-f4b6-480b-9545-569baf9670a8-istio-token\" (UniqueName: \"kubernetes.io/projected/db62d92e-f4b6-480b-9545-569baf9670a8-istio-token\") pod \"db62d92e-f4b6-480b-9545-569baf9670a8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " Oct 30 01:30:47.559404 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.559274 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-credential-socket\") pod \"db62d92e-f4b6-480b-9545-569baf9670a8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " Oct 30 01:30:47.559404 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.559330 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-workload-socket\") pod \"db62d92e-f4b6-480b-9545-569baf9670a8\" (UID: \"db62d92e-f4b6-480b-9545-569baf9670a8\") " Oct 30 01:30:47.559404 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.559368 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-workload-certs" pod "db62d92e-f4b6-480b-9545-569baf9670a8" (UID: "db62d92e-f4b6-480b-9545-569baf9670a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:47.559744 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.559495 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db62d92e-f4b6-480b-9545-569baf9670a8-istiod-ca-cert" pod "db62d92e-f4b6-480b-9545-569baf9670a8" (UID: "db62d92e-f4b6-480b-9545-569baf9670a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:47.559835 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.559746 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-workload-socket" pod "db62d92e-f4b6-480b-9545-569baf9670a8" (UID: "db62d92e-f4b6-480b-9545-569baf9670a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:47.560407 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.559937 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-credential-socket" pod "db62d92e-f4b6-480b-9545-569baf9670a8" (UID: "db62d92e-f4b6-480b-9545-569baf9670a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:47.560407 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.560070 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:47.560407 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.560091 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db62d92e-f4b6-480b-9545-569baf9670a8-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:47.560407 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.560142 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:47.560407 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.560155 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:47.562506 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.562435 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db62d92e-f4b6-480b-9545-569baf9670a8-istio-podinfo" pod "db62d92e-f4b6-480b-9545-569baf9670a8" (UID: "db62d92e-f4b6-480b-9545-569baf9670a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:30:47.562506 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.562435 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db62d92e-f4b6-480b-9545-569baf9670a8-istio-token" pod "db62d92e-f4b6-480b-9545-569baf9670a8" (UID: "db62d92e-f4b6-480b-9545-569baf9670a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:47.566505 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.566469 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-istio-envoy" pod "db62d92e-f4b6-480b-9545-569baf9670a8" (UID: "db62d92e-f4b6-480b-9545-569baf9670a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:47.567519 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.567482 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db62d92e-f4b6-480b-9545-569baf9670a8-kube-api-access-km6w9" pod "db62d92e-f4b6-480b-9545-569baf9670a8" (UID: "db62d92e-f4b6-480b-9545-569baf9670a8"). InnerVolumeSpecName "kube-api-access-km6w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:47.661910 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.661864 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db62d92e-f4b6-480b-9545-569baf9670a8-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:47.661910 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.661911 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km6w9\" (UniqueName: \"kubernetes.io/projected/db62d92e-f4b6-480b-9545-569baf9670a8-kube-api-access-km6w9\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:47.661910 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.661927 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:47.662229 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.661942 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db62d92e-f4b6-480b-9545-569baf9670a8-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:47.662229 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:47.661954 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db62d92e-f4b6-480b-9545-569baf9670a8-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:48.024303 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:48.024265 5300 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-206.ec2.internal" podUID="3b175f0644888f98ec9cdc35355eb1b8" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 01:30:48.334405 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:48.334370 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:48.487222 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:48.487190 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8" Oct 30 01:30:48.515489 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:48.515457 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:48.570407 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:48.570368 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8"] Oct 30 01:30:48.572876 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:48.572854 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d827b7-openshift-default-6b4c44bc5c-426h8"] Oct 30 01:30:48.795810 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:48.795755 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:49.604074 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:49.604035 5300 patch_prober.go:29] interesting pod/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:30:49.604479 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:49.604108 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" podUID="78eeb03d-9184-4a3d-af85-b8391731e828" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:30:49.823517 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:49.823478 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db62d92e-f4b6-480b-9545-569baf9670a8" path="/var/lib/kubelet/pods/db62d92e-f4b6-480b-9545-569baf9670a8/volumes" Oct 30 01:30:49.852652 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:49.852621 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:50.252797 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.252716 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 01:30:50.396855 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.396820 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78eeb03d-9184-4a3d-af85-b8391731e828-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78eeb03d-9184-4a3d-af85-b8391731e828-istiod-ca-cert\") pod \"78eeb03d-9184-4a3d-af85-b8391731e828\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " Oct 30 01:30:50.396855 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.396863 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78eeb03d-9184-4a3d-af85-b8391731e828-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78eeb03d-9184-4a3d-af85-b8391731e828-istio-podinfo\") pod \"78eeb03d-9184-4a3d-af85-b8391731e828\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " Oct 30 01:30:50.397132 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.396883 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78eeb03d-9184-4a3d-af85-b8391731e828-kube-api-access-tc5tb\" (UniqueName: \"kubernetes.io/projected/78eeb03d-9184-4a3d-af85-b8391731e828-kube-api-access-tc5tb\") pod \"78eeb03d-9184-4a3d-af85-b8391731e828\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " Oct 30 01:30:50.397132 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.396918 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78eeb03d-9184-4a3d-af85-b8391731e828-istio-token\" (UniqueName: \"kubernetes.io/projected/78eeb03d-9184-4a3d-af85-b8391731e828-istio-token\") pod \"78eeb03d-9184-4a3d-af85-b8391731e828\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " Oct 30 01:30:50.397132 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.396968 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-credential-socket\") pod \"78eeb03d-9184-4a3d-af85-b8391731e828\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " Oct 30 01:30:50.397132 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.397004 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-istio-envoy\") pod \"78eeb03d-9184-4a3d-af85-b8391731e828\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " Oct 30 01:30:50.397132 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.397041 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-workload-certs\") pod \"78eeb03d-9184-4a3d-af85-b8391731e828\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " Oct 30 01:30:50.397423 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.397169 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-istio-data\") pod \"78eeb03d-9184-4a3d-af85-b8391731e828\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " Oct 30 01:30:50.397423 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.397322 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-credential-socket" pod "78eeb03d-9184-4a3d-af85-b8391731e828" (UID: "78eeb03d-9184-4a3d-af85-b8391731e828"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:50.397423 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.397350 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-workload-socket\") pod \"78eeb03d-9184-4a3d-af85-b8391731e828\" (UID: \"78eeb03d-9184-4a3d-af85-b8391731e828\") " Oct 30 01:30:50.397423 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.397362 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78eeb03d-9184-4a3d-af85-b8391731e828-istiod-ca-cert" pod "78eeb03d-9184-4a3d-af85-b8391731e828" (UID: "78eeb03d-9184-4a3d-af85-b8391731e828"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:50.397636 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.397525 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-istio-data" pod "78eeb03d-9184-4a3d-af85-b8391731e828" (UID: "78eeb03d-9184-4a3d-af85-b8391731e828"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:50.397705 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.397635 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-workload-socket" pod "78eeb03d-9184-4a3d-af85-b8391731e828" (UID: "78eeb03d-9184-4a3d-af85-b8391731e828"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:50.397705 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.397654 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-workload-certs" pod "78eeb03d-9184-4a3d-af85-b8391731e828" (UID: "78eeb03d-9184-4a3d-af85-b8391731e828"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:50.397900 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.397884 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78eeb03d-9184-4a3d-af85-b8391731e828-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:50.397945 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.397904 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:50.397945 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.397914 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:50.397945 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.397923 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:50.397945 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.397931 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:50.399361 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.399332 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78eeb03d-9184-4a3d-af85-b8391731e828-istio-token" pod "78eeb03d-9184-4a3d-af85-b8391731e828" (UID: "78eeb03d-9184-4a3d-af85-b8391731e828"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:50.399663 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.399644 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78eeb03d-9184-4a3d-af85-b8391731e828-kube-api-access-tc5tb" pod "78eeb03d-9184-4a3d-af85-b8391731e828" (UID: "78eeb03d-9184-4a3d-af85-b8391731e828"). InnerVolumeSpecName "kube-api-access-tc5tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:50.399729 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.399691 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-istio-envoy" pod "78eeb03d-9184-4a3d-af85-b8391731e828" (UID: "78eeb03d-9184-4a3d-af85-b8391731e828"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:50.399729 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.399706 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78eeb03d-9184-4a3d-af85-b8391731e828-istio-podinfo" pod "78eeb03d-9184-4a3d-af85-b8391731e828" (UID: "78eeb03d-9184-4a3d-af85-b8391731e828"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:30:50.498738 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.498709 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78eeb03d-9184-4a3d-af85-b8391731e828-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:50.498738 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.498734 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc5tb\" (UniqueName: \"kubernetes.io/projected/78eeb03d-9184-4a3d-af85-b8391731e828-kube-api-access-tc5tb\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:50.498738 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.498744 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78eeb03d-9184-4a3d-af85-b8391731e828-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:50.498975 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.498753 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78eeb03d-9184-4a3d-af85-b8391731e828-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:50.504696 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.504618 5300 generic.go:356] "Generic (PLEG): container finished" podID="78eeb03d-9184-4a3d-af85-b8391731e828" containerID="2b4e8c4555bb286dae11f98457625a1e56a08fba51673d1c66aecdc08d31cbf8" exitCode=0 Oct 30 01:30:50.504696 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.504672 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" Oct 30 01:30:50.504903 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.504691 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" event={"ID":"78eeb03d-9184-4a3d-af85-b8391731e828","Type":"ContainerDied","Data":"2b4e8c4555bb286dae11f98457625a1e56a08fba51673d1c66aecdc08d31cbf8"} Oct 30 01:30:50.504903 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.504749 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv" event={"ID":"78eeb03d-9184-4a3d-af85-b8391731e828","Type":"ContainerDied","Data":"8191207da3a16a01ad6652c2c6160fb98000615332c24e3a25512d24931de1fb"} Oct 30 01:30:50.504903 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.504806 5300 scope.go:122] "RemoveContainer" containerID="2b4e8c4555bb286dae11f98457625a1e56a08fba51673d1c66aecdc08d31cbf8" Oct 30 01:30:50.515303 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.515277 5300 scope.go:122] "RemoveContainer" containerID="596071fa4f369aa0913c47126c60790e71a4f450fc817aab4668ab7904f8522d" Oct 30 01:30:50.524598 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.524578 5300 scope.go:122] "RemoveContainer" containerID="2b4e8c4555bb286dae11f98457625a1e56a08fba51673d1c66aecdc08d31cbf8" Oct 30 01:30:50.524866 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:30:50.524850 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b4e8c4555bb286dae11f98457625a1e56a08fba51673d1c66aecdc08d31cbf8\": container with ID starting with 2b4e8c4555bb286dae11f98457625a1e56a08fba51673d1c66aecdc08d31cbf8 not found: ID does not exist" containerID="2b4e8c4555bb286dae11f98457625a1e56a08fba51673d1c66aecdc08d31cbf8" Oct 30 01:30:50.524944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.524877 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b4e8c4555bb286dae11f98457625a1e56a08fba51673d1c66aecdc08d31cbf8"} err="failed to get container status \"2b4e8c4555bb286dae11f98457625a1e56a08fba51673d1c66aecdc08d31cbf8\": rpc error: code = NotFound desc = could not find container \"2b4e8c4555bb286dae11f98457625a1e56a08fba51673d1c66aecdc08d31cbf8\": container with ID starting with 2b4e8c4555bb286dae11f98457625a1e56a08fba51673d1c66aecdc08d31cbf8 not found: ID does not exist" Oct 30 01:30:50.524944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.524893 5300 scope.go:122] "RemoveContainer" containerID="596071fa4f369aa0913c47126c60790e71a4f450fc817aab4668ab7904f8522d" Oct 30 01:30:50.525161 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:30:50.525143 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"596071fa4f369aa0913c47126c60790e71a4f450fc817aab4668ab7904f8522d\": container with ID starting with 596071fa4f369aa0913c47126c60790e71a4f450fc817aab4668ab7904f8522d not found: ID does not exist" containerID="596071fa4f369aa0913c47126c60790e71a4f450fc817aab4668ab7904f8522d" Oct 30 01:30:50.525208 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.525168 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596071fa4f369aa0913c47126c60790e71a4f450fc817aab4668ab7904f8522d"} err="failed to get container status \"596071fa4f369aa0913c47126c60790e71a4f450fc817aab4668ab7904f8522d\": rpc error: code = NotFound desc = could not find container \"596071fa4f369aa0913c47126c60790e71a4f450fc817aab4668ab7904f8522d\": container with ID starting with 596071fa4f369aa0913c47126c60790e71a4f450fc817aab4668ab7904f8522d not found: ID does not exist" Oct 30 01:30:50.569561 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.569533 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv"] Oct 30 01:30:50.575773 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.575736 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c104b8-openshift-default-6fdc484f75-kb9nv"] Oct 30 01:30:50.638030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.637991 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:50.914644 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:50.914559 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:51.161782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:51.157845 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:51.594201 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:30:51.594160 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:30:51.829592 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:51.829554 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78eeb03d-9184-4a3d-af85-b8391731e828" path="/var/lib/kubelet/pods/78eeb03d-9184-4a3d-af85-b8391731e828/volumes" Oct 30 01:30:51.842555 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:51.842515 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:52.230389 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:52.230356 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:52.413877 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:52.413838 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:53.794699 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:53.794656 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:53.912110 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:53.912081 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:54.089416 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:54.089324 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp"] Oct 30 01:30:54.089644 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:54.089620 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" podUID="12e75f96-d218-400a-8b56-0cea344f3e41" containerName="istio-proxy" containerID="cri-o://1a0e1feca9b621ad561013ee0cccc5b6caa47cdb9b0fdd18af45b6e5c7b863a1" gracePeriod=30 Oct 30 01:30:54.210801 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:54.209591 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:56.027587 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:56.027540 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:56.684743 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:56.684711 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:57.751950 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:57.751916 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:59.361193 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.361136 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 01:30:59.513429 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.513398 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-workload-certs\") pod \"12e75f96-d218-400a-8b56-0cea344f3e41\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " Oct 30 01:30:59.513623 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.513450 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12e75f96-d218-400a-8b56-0cea344f3e41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12e75f96-d218-400a-8b56-0cea344f3e41-istio-podinfo\") pod \"12e75f96-d218-400a-8b56-0cea344f3e41\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " Oct 30 01:30:59.513623 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.513501 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-istio-data\") pod \"12e75f96-d218-400a-8b56-0cea344f3e41\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " Oct 30 01:30:59.513623 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.513534 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-credential-socket\") pod \"12e75f96-d218-400a-8b56-0cea344f3e41\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " Oct 30 01:30:59.513623 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.513606 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12e75f96-d218-400a-8b56-0cea344f3e41-istio-token\" (UniqueName: \"kubernetes.io/projected/12e75f96-d218-400a-8b56-0cea344f3e41-istio-token\") pod \"12e75f96-d218-400a-8b56-0cea344f3e41\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " Oct 30 01:30:59.513885 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.513674 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-workload-certs" pod "12e75f96-d218-400a-8b56-0cea344f3e41" (UID: "12e75f96-d218-400a-8b56-0cea344f3e41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:59.513885 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.513688 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-workload-socket\") pod \"12e75f96-d218-400a-8b56-0cea344f3e41\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " Oct 30 01:30:59.513885 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.513756 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12e75f96-d218-400a-8b56-0cea344f3e41-kube-api-access-ppdnf\" (UniqueName: \"kubernetes.io/projected/12e75f96-d218-400a-8b56-0cea344f3e41-kube-api-access-ppdnf\") pod \"12e75f96-d218-400a-8b56-0cea344f3e41\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " Oct 30 01:30:59.513885 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.513824 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12e75f96-d218-400a-8b56-0cea344f3e41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12e75f96-d218-400a-8b56-0cea344f3e41-istiod-ca-cert\") pod \"12e75f96-d218-400a-8b56-0cea344f3e41\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " Oct 30 01:30:59.513885 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.513851 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-istio-data" pod "12e75f96-d218-400a-8b56-0cea344f3e41" (UID: "12e75f96-d218-400a-8b56-0cea344f3e41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:59.514145 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.513888 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-istio-envoy\") pod \"12e75f96-d218-400a-8b56-0cea344f3e41\" (UID: \"12e75f96-d218-400a-8b56-0cea344f3e41\") " Oct 30 01:30:59.514145 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.513907 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-credential-socket" pod "12e75f96-d218-400a-8b56-0cea344f3e41" (UID: "12e75f96-d218-400a-8b56-0cea344f3e41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:59.514145 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.514005 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-workload-socket" pod "12e75f96-d218-400a-8b56-0cea344f3e41" (UID: "12e75f96-d218-400a-8b56-0cea344f3e41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:59.514314 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.514237 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12e75f96-d218-400a-8b56-0cea344f3e41-istiod-ca-cert" pod "12e75f96-d218-400a-8b56-0cea344f3e41" (UID: "12e75f96-d218-400a-8b56-0cea344f3e41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:59.515488 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.514529 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:59.515488 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.514556 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12e75f96-d218-400a-8b56-0cea344f3e41-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:59.515488 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.514568 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:59.515488 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.514580 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:59.515488 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.514588 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:59.516509 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.516483 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e75f96-d218-400a-8b56-0cea344f3e41-istio-token" pod "12e75f96-d218-400a-8b56-0cea344f3e41" (UID: "12e75f96-d218-400a-8b56-0cea344f3e41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:59.516642 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.516572 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e75f96-d218-400a-8b56-0cea344f3e41-kube-api-access-ppdnf" pod "12e75f96-d218-400a-8b56-0cea344f3e41" (UID: "12e75f96-d218-400a-8b56-0cea344f3e41"). InnerVolumeSpecName "kube-api-access-ppdnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:59.516642 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.516613 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12e75f96-d218-400a-8b56-0cea344f3e41-istio-podinfo" pod "12e75f96-d218-400a-8b56-0cea344f3e41" (UID: "12e75f96-d218-400a-8b56-0cea344f3e41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:30:59.516838 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.516657 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-istio-envoy" pod "12e75f96-d218-400a-8b56-0cea344f3e41" (UID: "12e75f96-d218-400a-8b56-0cea344f3e41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:30:59.561749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.561716 5300 generic.go:356] "Generic (PLEG): container finished" podID="12e75f96-d218-400a-8b56-0cea344f3e41" containerID="1a0e1feca9b621ad561013ee0cccc5b6caa47cdb9b0fdd18af45b6e5c7b863a1" exitCode=0 Oct 30 01:30:59.561926 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.561788 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" Oct 30 01:30:59.561926 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.561790 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" event={"ID":"12e75f96-d218-400a-8b56-0cea344f3e41","Type":"ContainerDied","Data":"1a0e1feca9b621ad561013ee0cccc5b6caa47cdb9b0fdd18af45b6e5c7b863a1"} Oct 30 01:30:59.561926 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.561887 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp" event={"ID":"12e75f96-d218-400a-8b56-0cea344f3e41","Type":"ContainerDied","Data":"6862b53370e72768353a5ac9d9b326420ee8ae0761bef77ad426c3b6b1091740"} Oct 30 01:30:59.561926 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.561919 5300 scope.go:122] "RemoveContainer" containerID="1a0e1feca9b621ad561013ee0cccc5b6caa47cdb9b0fdd18af45b6e5c7b863a1" Oct 30 01:30:59.572458 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.572442 5300 scope.go:122] "RemoveContainer" containerID="1a0e1feca9b621ad561013ee0cccc5b6caa47cdb9b0fdd18af45b6e5c7b863a1" Oct 30 01:30:59.572741 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:30:59.572718 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a0e1feca9b621ad561013ee0cccc5b6caa47cdb9b0fdd18af45b6e5c7b863a1\": container with ID starting with 1a0e1feca9b621ad561013ee0cccc5b6caa47cdb9b0fdd18af45b6e5c7b863a1 not found: ID does not exist" containerID="1a0e1feca9b621ad561013ee0cccc5b6caa47cdb9b0fdd18af45b6e5c7b863a1" Oct 30 01:30:59.572873 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.572753 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a0e1feca9b621ad561013ee0cccc5b6caa47cdb9b0fdd18af45b6e5c7b863a1"} err="failed to get container status \"1a0e1feca9b621ad561013ee0cccc5b6caa47cdb9b0fdd18af45b6e5c7b863a1\": rpc error: code = NotFound desc = could not find container \"1a0e1feca9b621ad561013ee0cccc5b6caa47cdb9b0fdd18af45b6e5c7b863a1\": container with ID starting with 1a0e1feca9b621ad561013ee0cccc5b6caa47cdb9b0fdd18af45b6e5c7b863a1 not found: ID does not exist" Oct 30 01:30:59.615541 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.615484 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12e75f96-d218-400a-8b56-0cea344f3e41-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:59.615541 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.615517 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e75f96-d218-400a-8b56-0cea344f3e41-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:59.615541 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.615533 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppdnf\" (UniqueName: \"kubernetes.io/projected/12e75f96-d218-400a-8b56-0cea344f3e41-kube-api-access-ppdnf\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:59.615743 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.615546 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12e75f96-d218-400a-8b56-0cea344f3e41-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:30:59.636517 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.636463 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp"] Oct 30 01:30:59.640590 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.640560 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dca38d-openshift-default-56494c5f9c-2s6sp"] Oct 30 01:30:59.670540 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.670506 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:59.822020 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.821987 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12e75f96-d218-400a-8b56-0cea344f3e41" path="/var/lib/kubelet/pods/12e75f96-d218-400a-8b56-0cea344f3e41/volumes" Oct 30 01:30:59.854987 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.854960 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh"] Oct 30 01:30:59.860391 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:30:59.860368 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm"] Oct 30 01:31:00.058049 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:00.058018 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:00.274796 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:00.274732 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:01.450976 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:01.450938 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:03.932242 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:03.932205 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:04.613425 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:04.613388 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:04.855782 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:04.855726 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh" podUID="31f807be-26ff-4f56-9e5b-626f349d6206" containerName="payload-processing" containerID="cri-o://e9766205cc9272a80f25c8a8c05fb7010231d22fb44440a4069c1c0c9c94b145" gracePeriod=25 Oct 30 01:31:04.861217 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:04.861185 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm" podUID="78aef360-fd10-4741-b251-30ab37a63a05" containerName="payload-pre-processing" containerID="cri-o://a6b4a575a52636ded792ccad52052967af4c870dca77316a061fce24d7a4aab4" gracePeriod=25 Oct 30 01:31:05.846086 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:05.846031 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:07.345474 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:07.345437 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn"] Oct 30 01:31:07.346003 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:07.345716 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn" podUID="a2c3a225-7324-4958-8533-29b9c9d2919b" containerName="maas-api" containerID="cri-o://a236a87ea9c490ffc8a4128a37c15839a9cee6573cb77cc8f006def80c46b661" gracePeriod=30 Oct 30 01:31:07.463145 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:07.463121 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:07.633262 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:07.632922 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn" Oct 30 01:31:07.640296 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:07.639657 5300 generic.go:356] "Generic (PLEG): container finished" podID="a2c3a225-7324-4958-8533-29b9c9d2919b" containerID="a236a87ea9c490ffc8a4128a37c15839a9cee6573cb77cc8f006def80c46b661" exitCode=0 Oct 30 01:31:07.640296 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:07.639902 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn" event={"ID":"a2c3a225-7324-4958-8533-29b9c9d2919b","Type":"ContainerDied","Data":"a236a87ea9c490ffc8a4128a37c15839a9cee6573cb77cc8f006def80c46b661"} Oct 30 01:31:07.640296 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:07.639954 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn" event={"ID":"a2c3a225-7324-4958-8533-29b9c9d2919b","Type":"ContainerDied","Data":"89d00bd1c5106be30ebe652ac755810c16d9f2395410e3e4c32c2e4795df02b1"} Oct 30 01:31:07.640296 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:07.639982 5300 scope.go:122] "RemoveContainer" containerID="a236a87ea9c490ffc8a4128a37c15839a9cee6573cb77cc8f006def80c46b661" Oct 30 01:31:07.640296 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:07.640128 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn" Oct 30 01:31:07.652541 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:07.652499 5300 scope.go:122] "RemoveContainer" containerID="a236a87ea9c490ffc8a4128a37c15839a9cee6573cb77cc8f006def80c46b661" Oct 30 01:31:07.652925 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:31:07.652897 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a236a87ea9c490ffc8a4128a37c15839a9cee6573cb77cc8f006def80c46b661\": container with ID starting with a236a87ea9c490ffc8a4128a37c15839a9cee6573cb77cc8f006def80c46b661 not found: ID does not exist" containerID="a236a87ea9c490ffc8a4128a37c15839a9cee6573cb77cc8f006def80c46b661" Oct 30 01:31:07.653029 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:07.652936 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a236a87ea9c490ffc8a4128a37c15839a9cee6573cb77cc8f006def80c46b661"} err="failed to get container status \"a236a87ea9c490ffc8a4128a37c15839a9cee6573cb77cc8f006def80c46b661\": rpc error: code = NotFound desc = could not find container \"a236a87ea9c490ffc8a4128a37c15839a9cee6573cb77cc8f006def80c46b661\": container with ID starting with a236a87ea9c490ffc8a4128a37c15839a9cee6573cb77cc8f006def80c46b661 not found: ID does not exist" Oct 30 01:31:07.711015 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:07.710984 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2c3a225-7324-4958-8533-29b9c9d2919b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c3a225-7324-4958-8533-29b9c9d2919b-maas-api-tls\") pod \"a2c3a225-7324-4958-8533-29b9c9d2919b\" (UID: \"a2c3a225-7324-4958-8533-29b9c9d2919b\") " Oct 30 01:31:07.711184 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:07.711038 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2c3a225-7324-4958-8533-29b9c9d2919b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c3a225-7324-4958-8533-29b9c9d2919b-openshift-service-ca\") pod \"a2c3a225-7324-4958-8533-29b9c9d2919b\" (UID: \"a2c3a225-7324-4958-8533-29b9c9d2919b\") " Oct 30 01:31:07.711184 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:07.711151 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2c3a225-7324-4958-8533-29b9c9d2919b-kube-api-access-mp4sw\" (UniqueName: \"kubernetes.io/projected/a2c3a225-7324-4958-8533-29b9c9d2919b-kube-api-access-mp4sw\") pod \"a2c3a225-7324-4958-8533-29b9c9d2919b\" (UID: \"a2c3a225-7324-4958-8533-29b9c9d2919b\") " Oct 30 01:31:07.711344 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:07.711309 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2c3a225-7324-4958-8533-29b9c9d2919b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c3a225-7324-4958-8533-29b9c9d2919b-metrics-certs\") pod \"a2c3a225-7324-4958-8533-29b9c9d2919b\" (UID: \"a2c3a225-7324-4958-8533-29b9c9d2919b\") " Oct 30 01:31:07.711507 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:07.711477 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c3a225-7324-4958-8533-29b9c9d2919b-openshift-service-ca" pod "a2c3a225-7324-4958-8533-29b9c9d2919b" (UID: "a2c3a225-7324-4958-8533-29b9c9d2919b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:31:07.711885 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:07.711859 5300 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c3a225-7324-4958-8533-29b9c9d2919b-openshift-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:07.713276 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:07.713247 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2c3a225-7324-4958-8533-29b9c9d2919b-maas-api-tls" pod "a2c3a225-7324-4958-8533-29b9c9d2919b" (UID: "a2c3a225-7324-4958-8533-29b9c9d2919b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:31:07.713838 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:07.713809 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2c3a225-7324-4958-8533-29b9c9d2919b-metrics-certs" pod "a2c3a225-7324-4958-8533-29b9c9d2919b" (UID: "a2c3a225-7324-4958-8533-29b9c9d2919b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:31:07.713934 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:07.713869 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c3a225-7324-4958-8533-29b9c9d2919b-kube-api-access-mp4sw" pod "a2c3a225-7324-4958-8533-29b9c9d2919b" (UID: "a2c3a225-7324-4958-8533-29b9c9d2919b"). InnerVolumeSpecName "kube-api-access-mp4sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:07.813169 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:07.813134 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp4sw\" (UniqueName: \"kubernetes.io/projected/a2c3a225-7324-4958-8533-29b9c9d2919b-kube-api-access-mp4sw\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:07.813169 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:07.813172 5300 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c3a225-7324-4958-8533-29b9c9d2919b-metrics-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:07.813484 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:07.813187 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c3a225-7324-4958-8533-29b9c9d2919b-maas-api-tls\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:07.985972 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:07.985938 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn"] Oct 30 01:31:07.999797 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:07.999768 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e4f8a6-6b84f9bc9d-ncvpn"] Oct 30 01:31:08.094040 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:08.094013 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:08.267527 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:08.267448 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh" podUID="31f807be-26ff-4f56-9e5b-626f349d6206" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:31:08.359993 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:08.359962 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:08.672367 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:08.672285 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:09.824434 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:09.824398 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2c3a225-7324-4958-8533-29b9c9d2919b" path="/var/lib/kubelet/pods/a2c3a225-7324-4958-8533-29b9c9d2919b/volumes" Oct 30 01:31:10.167705 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:10.167623 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:10.284749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:10.284712 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm" podUID="78aef360-fd10-4741-b251-30ab37a63a05" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:31:12.016649 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:12.016611 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:14.201053 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:14.201015 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:15.484260 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:15.484230 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:15.689006 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:15.688976 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:15.841496 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:15.841418 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:16.155205 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:16.155117 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:16.306120 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:16.306092 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:16.743234 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:16.743201 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:17.512828 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:17.512751 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:17.765686 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:17.765611 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:17.846034 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:17.845998 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb"] Oct 30 01:31:17.846291 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:17.846245 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" podUID="72cf531e-9c97-4a15-a1a4-0beb72c53371" containerName="istio-proxy" containerID="cri-o://9cc12a9ce996ac6873eecdaf89eaf3385e7e42a3deebbadedafc90aac161ac25" gracePeriod=30 Oct 30 01:31:17.956000 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:17.950402 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:18.201415 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:18.201335 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:18.268346 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:18.268305 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh" podUID="31f807be-26ff-4f56-9e5b-626f349d6206" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:31:20.149962 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:20.149933 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:20.285257 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:20.285224 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm" podUID="78aef360-fd10-4741-b251-30ab37a63a05" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:31:21.592553 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:31:21.592517 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:31:21.803055 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:21.803025 5300 scope.go:122] "RemoveContainer" containerID="e3f0420d312b02f08636fe9caf4ea8fb2200987eba800ca086ff703ebc5a6fa3" Oct 30 01:31:21.807554 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:21.807530 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:21.813386 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:21.813363 5300 scope.go:122] "RemoveContainer" containerID="981208c4e05bf33db5ad768118b71b789f8794874cea061663aa65a16be50af5" Oct 30 01:31:23.113319 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.113259 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:31:23.163674 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.163638 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:23.227094 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.227056 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-envoy\") pod \"72cf531e-9c97-4a15-a1a4-0beb72c53371\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " Oct 30 01:31:23.227275 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.227114 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-credential-socket\") pod \"72cf531e-9c97-4a15-a1a4-0beb72c53371\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " Oct 30 01:31:23.227275 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.227158 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-token\" (UniqueName: \"kubernetes.io/projected/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-token\") pod \"72cf531e-9c97-4a15-a1a4-0beb72c53371\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " Oct 30 01:31:23.227406 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.227272 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72cf531e-9c97-4a15-a1a4-0beb72c53371-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72cf531e-9c97-4a15-a1a4-0beb72c53371-istiod-ca-cert\") pod \"72cf531e-9c97-4a15-a1a4-0beb72c53371\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " Oct 30 01:31:23.227406 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.227338 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72cf531e-9c97-4a15-a1a4-0beb72c53371-kube-api-access-5486q\" (UniqueName: \"kubernetes.io/projected/72cf531e-9c97-4a15-a1a4-0beb72c53371-kube-api-access-5486q\") pod \"72cf531e-9c97-4a15-a1a4-0beb72c53371\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " Oct 30 01:31:23.227406 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.227376 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-data\") pod \"72cf531e-9c97-4a15-a1a4-0beb72c53371\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " Oct 30 01:31:23.227576 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.227430 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-credential-socket" pod "72cf531e-9c97-4a15-a1a4-0beb72c53371" (UID: "72cf531e-9c97-4a15-a1a4-0beb72c53371"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:23.227576 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.227447 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-workload-certs\") pod \"72cf531e-9c97-4a15-a1a4-0beb72c53371\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " Oct 30 01:31:23.227576 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.227549 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-workload-socket\") pod \"72cf531e-9c97-4a15-a1a4-0beb72c53371\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " Oct 30 01:31:23.227741 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.227611 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-podinfo\") pod \"72cf531e-9c97-4a15-a1a4-0beb72c53371\" (UID: \"72cf531e-9c97-4a15-a1a4-0beb72c53371\") " Oct 30 01:31:23.227741 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.227674 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-workload-certs" pod "72cf531e-9c97-4a15-a1a4-0beb72c53371" (UID: "72cf531e-9c97-4a15-a1a4-0beb72c53371"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:23.227892 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.227741 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-data" pod "72cf531e-9c97-4a15-a1a4-0beb72c53371" (UID: "72cf531e-9c97-4a15-a1a4-0beb72c53371"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:23.227956 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.227931 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-workload-socket" pod "72cf531e-9c97-4a15-a1a4-0beb72c53371" (UID: "72cf531e-9c97-4a15-a1a4-0beb72c53371"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:23.228235 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.228196 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72cf531e-9c97-4a15-a1a4-0beb72c53371-istiod-ca-cert" pod "72cf531e-9c97-4a15-a1a4-0beb72c53371" (UID: "72cf531e-9c97-4a15-a1a4-0beb72c53371"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:31:23.228354 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.228271 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:23.228354 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.228292 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:23.228354 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.228306 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:23.228354 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.228321 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:23.229977 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.229952 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-token" pod "72cf531e-9c97-4a15-a1a4-0beb72c53371" (UID: "72cf531e-9c97-4a15-a1a4-0beb72c53371"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:23.230084 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.230025 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-podinfo" pod "72cf531e-9c97-4a15-a1a4-0beb72c53371" (UID: "72cf531e-9c97-4a15-a1a4-0beb72c53371"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:31:23.230155 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.230125 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-envoy" pod "72cf531e-9c97-4a15-a1a4-0beb72c53371" (UID: "72cf531e-9c97-4a15-a1a4-0beb72c53371"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:23.230214 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.230190 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72cf531e-9c97-4a15-a1a4-0beb72c53371-kube-api-access-5486q" pod "72cf531e-9c97-4a15-a1a4-0beb72c53371" (UID: "72cf531e-9c97-4a15-a1a4-0beb72c53371"). InnerVolumeSpecName "kube-api-access-5486q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:23.296176 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.296150 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:23.329137 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.329110 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:23.329137 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.329133 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:23.329308 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.329143 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72cf531e-9c97-4a15-a1a4-0beb72c53371-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:23.329308 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.329150 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72cf531e-9c97-4a15-a1a4-0beb72c53371-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:23.329308 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.329159 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5486q\" (UniqueName: \"kubernetes.io/projected/72cf531e-9c97-4a15-a1a4-0beb72c53371-kube-api-access-5486q\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:23.524734 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.524700 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:23.647788 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.647719 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:23.756937 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.756899 5300 generic.go:356] "Generic (PLEG): container finished" podID="72cf531e-9c97-4a15-a1a4-0beb72c53371" containerID="9cc12a9ce996ac6873eecdaf89eaf3385e7e42a3deebbadedafc90aac161ac25" exitCode=0 Oct 30 01:31:23.757136 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.756954 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" Oct 30 01:31:23.757136 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.756965 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" event={"ID":"72cf531e-9c97-4a15-a1a4-0beb72c53371","Type":"ContainerDied","Data":"9cc12a9ce996ac6873eecdaf89eaf3385e7e42a3deebbadedafc90aac161ac25"} Oct 30 01:31:23.757136 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.757001 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb" event={"ID":"72cf531e-9c97-4a15-a1a4-0beb72c53371","Type":"ContainerDied","Data":"7fe0a8000980c2e8e7d1016ef693d7e36f47549b297d55816c7dc8cde1103773"} Oct 30 01:31:23.757136 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.757022 5300 scope.go:122] "RemoveContainer" containerID="9cc12a9ce996ac6873eecdaf89eaf3385e7e42a3deebbadedafc90aac161ac25" Oct 30 01:31:23.767610 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.767591 5300 scope.go:122] "RemoveContainer" containerID="6df25cbac4ebe899038b6e9cb904f9e50d6b8b0518d11ac5ed979b89f11b6f7d" Oct 30 01:31:23.776287 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.776270 5300 scope.go:122] "RemoveContainer" containerID="9cc12a9ce996ac6873eecdaf89eaf3385e7e42a3deebbadedafc90aac161ac25" Oct 30 01:31:23.776542 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:31:23.776521 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cc12a9ce996ac6873eecdaf89eaf3385e7e42a3deebbadedafc90aac161ac25\": container with ID starting with 9cc12a9ce996ac6873eecdaf89eaf3385e7e42a3deebbadedafc90aac161ac25 not found: ID does not exist" containerID="9cc12a9ce996ac6873eecdaf89eaf3385e7e42a3deebbadedafc90aac161ac25" Oct 30 01:31:23.776593 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.776550 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cc12a9ce996ac6873eecdaf89eaf3385e7e42a3deebbadedafc90aac161ac25"} err="failed to get container status \"9cc12a9ce996ac6873eecdaf89eaf3385e7e42a3deebbadedafc90aac161ac25\": rpc error: code = NotFound desc = could not find container \"9cc12a9ce996ac6873eecdaf89eaf3385e7e42a3deebbadedafc90aac161ac25\": container with ID starting with 9cc12a9ce996ac6873eecdaf89eaf3385e7e42a3deebbadedafc90aac161ac25 not found: ID does not exist" Oct 30 01:31:23.776593 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.776567 5300 scope.go:122] "RemoveContainer" containerID="6df25cbac4ebe899038b6e9cb904f9e50d6b8b0518d11ac5ed979b89f11b6f7d" Oct 30 01:31:23.776849 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:31:23.776816 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6df25cbac4ebe899038b6e9cb904f9e50d6b8b0518d11ac5ed979b89f11b6f7d\": container with ID starting with 6df25cbac4ebe899038b6e9cb904f9e50d6b8b0518d11ac5ed979b89f11b6f7d not found: ID does not exist" containerID="6df25cbac4ebe899038b6e9cb904f9e50d6b8b0518d11ac5ed979b89f11b6f7d" Oct 30 01:31:23.776924 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.776854 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df25cbac4ebe899038b6e9cb904f9e50d6b8b0518d11ac5ed979b89f11b6f7d"} err="failed to get container status \"6df25cbac4ebe899038b6e9cb904f9e50d6b8b0518d11ac5ed979b89f11b6f7d\": rpc error: code = NotFound desc = could not find container \"6df25cbac4ebe899038b6e9cb904f9e50d6b8b0518d11ac5ed979b89f11b6f7d\": container with ID starting with 6df25cbac4ebe899038b6e9cb904f9e50d6b8b0518d11ac5ed979b89f11b6f7d not found: ID does not exist" Oct 30 01:31:23.832125 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.832096 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb"] Oct 30 01:31:23.846169 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:23.846142 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e4f8a6-openshift-default-565bc7bbfc-4lgfb"] Oct 30 01:31:25.126570 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.126502 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh" Oct 30 01:31:25.149161 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.149123 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31f807be-26ff-4f56-9e5b-626f349d6206-kube-api-access-xx6kk\" (UniqueName: \"kubernetes.io/projected/31f807be-26ff-4f56-9e5b-626f349d6206-kube-api-access-xx6kk\") pod \"31f807be-26ff-4f56-9e5b-626f349d6206\" (UID: \"31f807be-26ff-4f56-9e5b-626f349d6206\") " Oct 30 01:31:25.149324 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.149200 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31f807be-26ff-4f56-9e5b-626f349d6206-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f807be-26ff-4f56-9e5b-626f349d6206-plugins-config-volume\") pod \"31f807be-26ff-4f56-9e5b-626f349d6206\" (UID: \"31f807be-26ff-4f56-9e5b-626f349d6206\") " Oct 30 01:31:25.149562 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.149539 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f807be-26ff-4f56-9e5b-626f349d6206-plugins-config-volume" pod "31f807be-26ff-4f56-9e5b-626f349d6206" (UID: "31f807be-26ff-4f56-9e5b-626f349d6206"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:31:25.149796 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.149776 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f807be-26ff-4f56-9e5b-626f349d6206-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:25.151262 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.151236 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f807be-26ff-4f56-9e5b-626f349d6206-kube-api-access-xx6kk" pod "31f807be-26ff-4f56-9e5b-626f349d6206" (UID: "31f807be-26ff-4f56-9e5b-626f349d6206"). InnerVolumeSpecName "kube-api-access-xx6kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:25.170248 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.170230 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm" Oct 30 01:31:25.250327 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.250295 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78aef360-fd10-4741-b251-30ab37a63a05-kube-api-access-tl8sh\" (UniqueName: \"kubernetes.io/projected/78aef360-fd10-4741-b251-30ab37a63a05-kube-api-access-tl8sh\") pod \"78aef360-fd10-4741-b251-30ab37a63a05\" (UID: \"78aef360-fd10-4741-b251-30ab37a63a05\") " Oct 30 01:31:25.250541 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.250392 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78aef360-fd10-4741-b251-30ab37a63a05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78aef360-fd10-4741-b251-30ab37a63a05-plugins-config-volume\") pod \"78aef360-fd10-4741-b251-30ab37a63a05\" (UID: \"78aef360-fd10-4741-b251-30ab37a63a05\") " Oct 30 01:31:25.250811 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.250794 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx6kk\" (UniqueName: \"kubernetes.io/projected/31f807be-26ff-4f56-9e5b-626f349d6206-kube-api-access-xx6kk\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:25.250930 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.250811 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78aef360-fd10-4741-b251-30ab37a63a05-plugins-config-volume" pod "78aef360-fd10-4741-b251-30ab37a63a05" (UID: "78aef360-fd10-4741-b251-30ab37a63a05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:31:25.252489 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.252467 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78aef360-fd10-4741-b251-30ab37a63a05-kube-api-access-tl8sh" pod "78aef360-fd10-4741-b251-30ab37a63a05" (UID: "78aef360-fd10-4741-b251-30ab37a63a05"). InnerVolumeSpecName "kube-api-access-tl8sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:25.351965 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.351931 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78aef360-fd10-4741-b251-30ab37a63a05-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:25.351965 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.351960 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl8sh\" (UniqueName: \"kubernetes.io/projected/78aef360-fd10-4741-b251-30ab37a63a05-kube-api-access-tl8sh\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:25.456914 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.456831 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:25.770229 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.770193 5300 generic.go:356] "Generic (PLEG): container finished" podID="78aef360-fd10-4741-b251-30ab37a63a05" containerID="a6b4a575a52636ded792ccad52052967af4c870dca77316a061fce24d7a4aab4" exitCode=0 Oct 30 01:31:25.770410 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.770244 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm" Oct 30 01:31:25.770410 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.770273 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm" event={"ID":"78aef360-fd10-4741-b251-30ab37a63a05","Type":"ContainerDied","Data":"a6b4a575a52636ded792ccad52052967af4c870dca77316a061fce24d7a4aab4"} Oct 30 01:31:25.770410 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.770326 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm" event={"ID":"78aef360-fd10-4741-b251-30ab37a63a05","Type":"ContainerDied","Data":"2f6c680efdf6b1b5d72b65d8856a9307cf0b0708a71a520f63de39fa2a9700ff"} Oct 30 01:31:25.770410 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.770349 5300 scope.go:122] "RemoveContainer" containerID="a6b4a575a52636ded792ccad52052967af4c870dca77316a061fce24d7a4aab4" Oct 30 01:31:25.771748 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.771723 5300 generic.go:356] "Generic (PLEG): container finished" podID="31f807be-26ff-4f56-9e5b-626f349d6206" containerID="e9766205cc9272a80f25c8a8c05fb7010231d22fb44440a4069c1c0c9c94b145" exitCode=0 Oct 30 01:31:25.771892 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.771796 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh" Oct 30 01:31:25.771892 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.771806 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh" event={"ID":"31f807be-26ff-4f56-9e5b-626f349d6206","Type":"ContainerDied","Data":"e9766205cc9272a80f25c8a8c05fb7010231d22fb44440a4069c1c0c9c94b145"} Oct 30 01:31:25.771892 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.771859 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh" event={"ID":"31f807be-26ff-4f56-9e5b-626f349d6206","Type":"ContainerDied","Data":"bd654e01c9022bb0effc2bb517c6820022e514fc8aa33f9458268ad5bb4baee5"} Oct 30 01:31:25.781793 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.781663 5300 scope.go:122] "RemoveContainer" containerID="a6b4a575a52636ded792ccad52052967af4c870dca77316a061fce24d7a4aab4" Oct 30 01:31:25.782045 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:31:25.782015 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b4a575a52636ded792ccad52052967af4c870dca77316a061fce24d7a4aab4\": container with ID starting with a6b4a575a52636ded792ccad52052967af4c870dca77316a061fce24d7a4aab4 not found: ID does not exist" containerID="a6b4a575a52636ded792ccad52052967af4c870dca77316a061fce24d7a4aab4" Oct 30 01:31:25.782127 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.782053 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b4a575a52636ded792ccad52052967af4c870dca77316a061fce24d7a4aab4"} err="failed to get container status \"a6b4a575a52636ded792ccad52052967af4c870dca77316a061fce24d7a4aab4\": rpc error: code = NotFound desc = could not find container \"a6b4a575a52636ded792ccad52052967af4c870dca77316a061fce24d7a4aab4\": container with ID starting with a6b4a575a52636ded792ccad52052967af4c870dca77316a061fce24d7a4aab4 not found: ID does not exist" Oct 30 01:31:25.782127 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.782077 5300 scope.go:122] "RemoveContainer" containerID="e9766205cc9272a80f25c8a8c05fb7010231d22fb44440a4069c1c0c9c94b145" Oct 30 01:31:25.790591 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.790574 5300 scope.go:122] "RemoveContainer" containerID="e9766205cc9272a80f25c8a8c05fb7010231d22fb44440a4069c1c0c9c94b145" Oct 30 01:31:25.790872 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:31:25.790855 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9766205cc9272a80f25c8a8c05fb7010231d22fb44440a4069c1c0c9c94b145\": container with ID starting with e9766205cc9272a80f25c8a8c05fb7010231d22fb44440a4069c1c0c9c94b145 not found: ID does not exist" containerID="e9766205cc9272a80f25c8a8c05fb7010231d22fb44440a4069c1c0c9c94b145" Oct 30 01:31:25.790938 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.790878 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9766205cc9272a80f25c8a8c05fb7010231d22fb44440a4069c1c0c9c94b145"} err="failed to get container status \"e9766205cc9272a80f25c8a8c05fb7010231d22fb44440a4069c1c0c9c94b145\": rpc error: code = NotFound desc = could not find container \"e9766205cc9272a80f25c8a8c05fb7010231d22fb44440a4069c1c0c9c94b145\": container with ID starting with e9766205cc9272a80f25c8a8c05fb7010231d22fb44440a4069c1c0c9c94b145 not found: ID does not exist" Oct 30 01:31:25.820503 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.820477 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72cf531e-9c97-4a15-a1a4-0beb72c53371" path="/var/lib/kubelet/pods/72cf531e-9c97-4a15-a1a4-0beb72c53371/volumes" Oct 30 01:31:25.829217 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.829192 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh"] Oct 30 01:31:25.845403 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.845375 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e4f8a6-5cbc554998-npwhh"] Oct 30 01:31:25.900536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.900510 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm"] Oct 30 01:31:25.907864 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:25.907843 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4f8a6-64b7df794c-26vvm"] Oct 30 01:31:27.113872 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:27.113824 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:27.821285 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:27.821254 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31f807be-26ff-4f56-9e5b-626f349d6206" path="/var/lib/kubelet/pods/31f807be-26ff-4f56-9e5b-626f349d6206/volumes" Oct 30 01:31:27.821669 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:27.821653 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78aef360-fd10-4741-b251-30ab37a63a05" path="/var/lib/kubelet/pods/78aef360-fd10-4741-b251-30ab37a63a05/volumes" Oct 30 01:31:29.022710 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:29.022678 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:29.194559 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:29.194528 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:30.653238 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:30.653206 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:32.200931 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:32.200895 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:33.819541 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:33.819498 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:34.353391 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:34.353355 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:34.523152 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:34.523117 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:34.653114 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:34.653024 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:34.799819 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:34.799778 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:34.952124 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:34.952035 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:35.715863 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:35.715826 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:36.151254 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:36.151202 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:36.640441 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:36.640404 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:36.770465 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:36.770412 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:36.884501 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:36.884446 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:37.437601 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:37.437568 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:38.641292 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:38.641260 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:40.707724 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:40.707688 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:42.431783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:42.428517 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:43.670423 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:43.670387 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:43.857057 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:43.857010 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:44.156341 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:44.156306 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq"] Oct 30 01:31:44.156592 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:44.156556 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" podUID="cfadd1c5-3001-4ebb-8156-46628aa2e722" containerName="istio-proxy" containerID="cri-o://49d0fc62354fd7ecf47bf54b59cf530f3b55936f6b95a5035ffa577fc90c4921" gracePeriod=30 Oct 30 01:31:44.169899 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:44.169870 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx"] Oct 30 01:31:44.170080 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:44.170059 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" podUID="24445fd3-1ef0-4523-9548-7d1400bcb32b" containerName="istio-proxy" containerID="cri-o://1c809ade95391c4ccf74e39c84d8c0d7db863564a3e391a193368d41af1da592" gracePeriod=30 Oct 30 01:31:44.200420 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:44.200380 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl"] Oct 30 01:31:44.200600 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:44.200582 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" podUID="508889de-4033-41dd-9b1a-88925f799310" containerName="istio-proxy" containerID="cri-o://c2b85899622be1e89574dfb021893a6f5c66d0fc7a6167fcd129ac09aa300a60" gracePeriod=30 Oct 30 01:31:44.274425 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:44.274375 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:44.468358 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:44.468247 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:44.637983 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:44.637940 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:45.998738 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:45.998703 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:46.100541 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:46.100500 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:46.290993 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:46.290907 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:46.343164 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:46.343131 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6"] Oct 30 01:31:46.343417 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:46.343381 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" podUID="f79971c3-5d7b-4fd1-a84f-46b2c0190af6" containerName="istio-proxy" containerID="cri-o://1a5d65f474e166141d53f1499262266ca0ca1fddece615a60b1b2b9a8a1c8945" gracePeriod=30 Oct 30 01:31:46.409838 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:46.409802 5300 patch_prober.go:29] interesting pod/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:31:46.410010 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:46.409869 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" podUID="f79971c3-5d7b-4fd1-a84f-46b2c0190af6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:31:46.441502 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:46.441463 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:48.074299 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:48.074260 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:49.473201 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.472859 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:31:49.499657 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.499624 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:31:49.551862 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.551836 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:31:49.562263 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.562236 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms"] Oct 30 01:31:49.656645 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.656609 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-credential-socket\") pod \"24445fd3-1ef0-4523-9548-7d1400bcb32b\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " Oct 30 01:31:49.656856 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.656662 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-data\") pod \"cfadd1c5-3001-4ebb-8156-46628aa2e722\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " Oct 30 01:31:49.656856 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.656708 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24445fd3-1ef0-4523-9548-7d1400bcb32b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24445fd3-1ef0-4523-9548-7d1400bcb32b-istiod-ca-cert\") pod \"24445fd3-1ef0-4523-9548-7d1400bcb32b\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " Oct 30 01:31:49.656856 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.656783 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-workload-certs\") pod \"24445fd3-1ef0-4523-9548-7d1400bcb32b\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " Oct 30 01:31:49.656856 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.656828 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-istio-envoy\") pod \"508889de-4033-41dd-9b1a-88925f799310\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " Oct 30 01:31:49.657114 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.656862 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-workload-socket\") pod \"cfadd1c5-3001-4ebb-8156-46628aa2e722\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " Oct 30 01:31:49.657114 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.656900 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24445fd3-1ef0-4523-9548-7d1400bcb32b-kube-api-access-dg5hh\" (UniqueName: \"kubernetes.io/projected/24445fd3-1ef0-4523-9548-7d1400bcb32b-kube-api-access-dg5hh\") pod \"24445fd3-1ef0-4523-9548-7d1400bcb32b\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " Oct 30 01:31:49.657114 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.656936 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-workload-certs\") pod \"cfadd1c5-3001-4ebb-8156-46628aa2e722\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " Oct 30 01:31:49.657114 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.656937 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-credential-socket" pod "24445fd3-1ef0-4523-9548-7d1400bcb32b" (UID: "24445fd3-1ef0-4523-9548-7d1400bcb32b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.657114 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.656973 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-credential-socket\") pod \"508889de-4033-41dd-9b1a-88925f799310\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " Oct 30 01:31:49.657114 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.657005 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-workload-socket\") pod \"24445fd3-1ef0-4523-9548-7d1400bcb32b\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " Oct 30 01:31:49.657114 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.657036 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/508889de-4033-41dd-9b1a-88925f799310-istio-token\" (UniqueName: \"kubernetes.io/projected/508889de-4033-41dd-9b1a-88925f799310-istio-token\") pod \"508889de-4033-41dd-9b1a-88925f799310\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " Oct 30 01:31:49.657114 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.657081 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/508889de-4033-41dd-9b1a-88925f799310-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/508889de-4033-41dd-9b1a-88925f799310-istiod-ca-cert\") pod \"508889de-4033-41dd-9b1a-88925f799310\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " Oct 30 01:31:49.657114 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.657114 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-token\" (UniqueName: \"kubernetes.io/projected/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-token\") pod \"cfadd1c5-3001-4ebb-8156-46628aa2e722\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " Oct 30 01:31:49.657594 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.657151 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-workload-certs\") pod \"508889de-4033-41dd-9b1a-88925f799310\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " Oct 30 01:31:49.657594 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.657167 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24445fd3-1ef0-4523-9548-7d1400bcb32b-istiod-ca-cert" pod "24445fd3-1ef0-4523-9548-7d1400bcb32b" (UID: "24445fd3-1ef0-4523-9548-7d1400bcb32b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:31:49.657594 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.657183 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-envoy\") pod \"24445fd3-1ef0-4523-9548-7d1400bcb32b\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " Oct 30 01:31:49.657594 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.657211 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-credential-socket\") pod \"cfadd1c5-3001-4ebb-8156-46628aa2e722\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " Oct 30 01:31:49.657594 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.657244 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-istio-data\") pod \"508889de-4033-41dd-9b1a-88925f799310\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " Oct 30 01:31:49.657594 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.657278 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfadd1c5-3001-4ebb-8156-46628aa2e722-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfadd1c5-3001-4ebb-8156-46628aa2e722-istiod-ca-cert\") pod \"cfadd1c5-3001-4ebb-8156-46628aa2e722\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " Oct 30 01:31:49.657594 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.657310 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/508889de-4033-41dd-9b1a-88925f799310-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/508889de-4033-41dd-9b1a-88925f799310-istio-podinfo\") pod \"508889de-4033-41dd-9b1a-88925f799310\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " Oct 30 01:31:49.657594 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.657346 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-podinfo\") pod \"cfadd1c5-3001-4ebb-8156-46628aa2e722\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " Oct 30 01:31:49.657594 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.657382 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfadd1c5-3001-4ebb-8156-46628aa2e722-kube-api-access-xskkz\" (UniqueName: \"kubernetes.io/projected/cfadd1c5-3001-4ebb-8156-46628aa2e722-kube-api-access-xskkz\") pod \"cfadd1c5-3001-4ebb-8156-46628aa2e722\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " Oct 30 01:31:49.657594 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.657392 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-workload-certs" pod "24445fd3-1ef0-4523-9548-7d1400bcb32b" (UID: "24445fd3-1ef0-4523-9548-7d1400bcb32b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.657594 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.657430 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-podinfo\") pod \"24445fd3-1ef0-4523-9548-7d1400bcb32b\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " Oct 30 01:31:49.657594 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.657465 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-token\" (UniqueName: \"kubernetes.io/projected/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-token\") pod \"24445fd3-1ef0-4523-9548-7d1400bcb32b\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " Oct 30 01:31:49.657594 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.657465 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-workload-socket" pod "24445fd3-1ef0-4523-9548-7d1400bcb32b" (UID: "24445fd3-1ef0-4523-9548-7d1400bcb32b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.657594 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.657521 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/508889de-4033-41dd-9b1a-88925f799310-kube-api-access-746b9\" (UniqueName: \"kubernetes.io/projected/508889de-4033-41dd-9b1a-88925f799310-kube-api-access-746b9\") pod \"508889de-4033-41dd-9b1a-88925f799310\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " Oct 30 01:31:49.657594 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.657553 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-envoy\") pod \"cfadd1c5-3001-4ebb-8156-46628aa2e722\" (UID: \"cfadd1c5-3001-4ebb-8156-46628aa2e722\") " Oct 30 01:31:49.658465 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.657687 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-workload-socket" pod "cfadd1c5-3001-4ebb-8156-46628aa2e722" (UID: "cfadd1c5-3001-4ebb-8156-46628aa2e722"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.658465 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.657697 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-data\") pod \"24445fd3-1ef0-4523-9548-7d1400bcb32b\" (UID: \"24445fd3-1ef0-4523-9548-7d1400bcb32b\") " Oct 30 01:31:49.658465 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.657818 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-workload-socket\") pod \"508889de-4033-41dd-9b1a-88925f799310\" (UID: \"508889de-4033-41dd-9b1a-88925f799310\") " Oct 30 01:31:49.658642 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.658520 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/508889de-4033-41dd-9b1a-88925f799310-istiod-ca-cert" pod "508889de-4033-41dd-9b1a-88925f799310" (UID: "508889de-4033-41dd-9b1a-88925f799310"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:31:49.658979 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.658955 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.659076 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.658984 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/508889de-4033-41dd-9b1a-88925f799310-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.659076 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.659000 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.659076 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.659014 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24445fd3-1ef0-4523-9548-7d1400bcb32b-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.659076 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.659028 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.659076 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.659042 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.659394 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.659372 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-data" pod "cfadd1c5-3001-4ebb-8156-46628aa2e722" (UID: "cfadd1c5-3001-4ebb-8156-46628aa2e722"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.660105 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.659753 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-workload-certs" pod "508889de-4033-41dd-9b1a-88925f799310" (UID: "508889de-4033-41dd-9b1a-88925f799310"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.660105 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.659887 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-credential-socket" pod "508889de-4033-41dd-9b1a-88925f799310" (UID: "508889de-4033-41dd-9b1a-88925f799310"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.660105 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.660043 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-data" pod "24445fd3-1ef0-4523-9548-7d1400bcb32b" (UID: "24445fd3-1ef0-4523-9548-7d1400bcb32b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.660105 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.660067 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-workload-socket" pod "508889de-4033-41dd-9b1a-88925f799310" (UID: "508889de-4033-41dd-9b1a-88925f799310"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.660378 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.660275 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-istio-data" pod "508889de-4033-41dd-9b1a-88925f799310" (UID: "508889de-4033-41dd-9b1a-88925f799310"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.660853 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.660798 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-credential-socket" pod "cfadd1c5-3001-4ebb-8156-46628aa2e722" (UID: "cfadd1c5-3001-4ebb-8156-46628aa2e722"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.660853 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.660829 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-workload-certs" pod "cfadd1c5-3001-4ebb-8156-46628aa2e722" (UID: "cfadd1c5-3001-4ebb-8156-46628aa2e722"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.661126 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.661102 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-envoy" pod "24445fd3-1ef0-4523-9548-7d1400bcb32b" (UID: "24445fd3-1ef0-4523-9548-7d1400bcb32b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.661205 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.661152 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-istio-envoy" pod "508889de-4033-41dd-9b1a-88925f799310" (UID: "508889de-4033-41dd-9b1a-88925f799310"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.662191 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.662165 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-podinfo" pod "24445fd3-1ef0-4523-9548-7d1400bcb32b" (UID: "24445fd3-1ef0-4523-9548-7d1400bcb32b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:31:49.662191 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.662181 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/508889de-4033-41dd-9b1a-88925f799310-istio-token" pod "508889de-4033-41dd-9b1a-88925f799310" (UID: "508889de-4033-41dd-9b1a-88925f799310"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:49.662655 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.662628 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-podinfo" pod "cfadd1c5-3001-4ebb-8156-46628aa2e722" (UID: "cfadd1c5-3001-4ebb-8156-46628aa2e722"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:31:49.663009 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.662970 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfadd1c5-3001-4ebb-8156-46628aa2e722-istiod-ca-cert" pod "cfadd1c5-3001-4ebb-8156-46628aa2e722" (UID: "cfadd1c5-3001-4ebb-8156-46628aa2e722"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:31:49.663207 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.663184 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/508889de-4033-41dd-9b1a-88925f799310-istio-podinfo" pod "508889de-4033-41dd-9b1a-88925f799310" (UID: "508889de-4033-41dd-9b1a-88925f799310"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:31:49.663401 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.663383 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-envoy" pod "cfadd1c5-3001-4ebb-8156-46628aa2e722" (UID: "cfadd1c5-3001-4ebb-8156-46628aa2e722"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:49.663717 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.663702 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-token" pod "24445fd3-1ef0-4523-9548-7d1400bcb32b" (UID: "24445fd3-1ef0-4523-9548-7d1400bcb32b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:49.663895 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.663881 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24445fd3-1ef0-4523-9548-7d1400bcb32b-kube-api-access-dg5hh" pod "24445fd3-1ef0-4523-9548-7d1400bcb32b" (UID: "24445fd3-1ef0-4523-9548-7d1400bcb32b"). InnerVolumeSpecName "kube-api-access-dg5hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:49.664278 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.664263 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/508889de-4033-41dd-9b1a-88925f799310-kube-api-access-746b9" pod "508889de-4033-41dd-9b1a-88925f799310" (UID: "508889de-4033-41dd-9b1a-88925f799310"). InnerVolumeSpecName "kube-api-access-746b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:49.664521 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.664495 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfadd1c5-3001-4ebb-8156-46628aa2e722-kube-api-access-xskkz" pod "cfadd1c5-3001-4ebb-8156-46628aa2e722" (UID: "cfadd1c5-3001-4ebb-8156-46628aa2e722"). InnerVolumeSpecName "kube-api-access-xskkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:49.665214 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.665191 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-token" pod "cfadd1c5-3001-4ebb-8156-46628aa2e722" (UID: "cfadd1c5-3001-4ebb-8156-46628aa2e722"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:49.690371 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.690325 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:49.760136 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.760098 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.760136 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.760128 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.760136 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.760141 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.760437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.760153 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.760437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.760166 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfadd1c5-3001-4ebb-8156-46628aa2e722-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.760437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.760177 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/508889de-4033-41dd-9b1a-88925f799310-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.760437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.760187 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.760437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.760200 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xskkz\" (UniqueName: \"kubernetes.io/projected/cfadd1c5-3001-4ebb-8156-46628aa2e722-kube-api-access-xskkz\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.760437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.760212 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.760437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.760235 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.760437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.760247 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-746b9\" (UniqueName: \"kubernetes.io/projected/508889de-4033-41dd-9b1a-88925f799310-kube-api-access-746b9\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.760437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.760259 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.760437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.760272 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24445fd3-1ef0-4523-9548-7d1400bcb32b-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.760437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.760284 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.760437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.760295 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.760437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.760308 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.760437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.760319 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg5hh\" (UniqueName: \"kubernetes.io/projected/24445fd3-1ef0-4523-9548-7d1400bcb32b-kube-api-access-dg5hh\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.760437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.760331 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfadd1c5-3001-4ebb-8156-46628aa2e722-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.760437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.760343 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/508889de-4033-41dd-9b1a-88925f799310-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.760437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.760356 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/508889de-4033-41dd-9b1a-88925f799310-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.760437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.760369 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfadd1c5-3001-4ebb-8156-46628aa2e722-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:49.905572 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.905535 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:49.905783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.905633 5300 generic.go:356] "Generic (PLEG): container finished" podID="cfadd1c5-3001-4ebb-8156-46628aa2e722" containerID="49d0fc62354fd7ecf47bf54b59cf530f3b55936f6b95a5035ffa577fc90c4921" exitCode=0 Oct 30 01:31:49.905783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.905716 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" Oct 30 01:31:49.905783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.905717 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" event={"ID":"cfadd1c5-3001-4ebb-8156-46628aa2e722","Type":"ContainerDied","Data":"49d0fc62354fd7ecf47bf54b59cf530f3b55936f6b95a5035ffa577fc90c4921"} Oct 30 01:31:49.905960 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.905826 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq" event={"ID":"cfadd1c5-3001-4ebb-8156-46628aa2e722","Type":"ContainerDied","Data":"d6444f7a76adeadf0c0d60c826bb5d066086a791be7cf3274b328e264dce3e8d"} Oct 30 01:31:49.905960 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.905866 5300 scope.go:122] "RemoveContainer" containerID="49d0fc62354fd7ecf47bf54b59cf530f3b55936f6b95a5035ffa577fc90c4921" Oct 30 01:31:49.907250 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.907228 5300 generic.go:356] "Generic (PLEG): container finished" podID="24445fd3-1ef0-4523-9548-7d1400bcb32b" containerID="1c809ade95391c4ccf74e39c84d8c0d7db863564a3e391a193368d41af1da592" exitCode=0 Oct 30 01:31:49.907370 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.907284 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" Oct 30 01:31:49.907370 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.907318 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" event={"ID":"24445fd3-1ef0-4523-9548-7d1400bcb32b","Type":"ContainerDied","Data":"1c809ade95391c4ccf74e39c84d8c0d7db863564a3e391a193368d41af1da592"} Oct 30 01:31:49.907370 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.907363 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx" event={"ID":"24445fd3-1ef0-4523-9548-7d1400bcb32b","Type":"ContainerDied","Data":"b13f880b6478fa435561223cdf0597ec54701a1c7d5d21eaa3b63e97db1be3bb"} Oct 30 01:31:49.909060 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.909036 5300 generic.go:356] "Generic (PLEG): container finished" podID="508889de-4033-41dd-9b1a-88925f799310" containerID="c2b85899622be1e89574dfb021893a6f5c66d0fc7a6167fcd129ac09aa300a60" exitCode=0 Oct 30 01:31:49.909185 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.909086 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" Oct 30 01:31:49.909185 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.909124 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" event={"ID":"508889de-4033-41dd-9b1a-88925f799310","Type":"ContainerDied","Data":"c2b85899622be1e89574dfb021893a6f5c66d0fc7a6167fcd129ac09aa300a60"} Oct 30 01:31:49.909185 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.909171 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl" event={"ID":"508889de-4033-41dd-9b1a-88925f799310","Type":"ContainerDied","Data":"a03cae68de7307249d2779de88c448359893897d4375d7a222997dd3e187992e"} Oct 30 01:31:49.916231 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.916214 5300 scope.go:122] "RemoveContainer" containerID="49d0fc62354fd7ecf47bf54b59cf530f3b55936f6b95a5035ffa577fc90c4921" Oct 30 01:31:49.916521 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:31:49.916502 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49d0fc62354fd7ecf47bf54b59cf530f3b55936f6b95a5035ffa577fc90c4921\": container with ID starting with 49d0fc62354fd7ecf47bf54b59cf530f3b55936f6b95a5035ffa577fc90c4921 not found: ID does not exist" containerID="49d0fc62354fd7ecf47bf54b59cf530f3b55936f6b95a5035ffa577fc90c4921" Oct 30 01:31:49.916565 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.916530 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49d0fc62354fd7ecf47bf54b59cf530f3b55936f6b95a5035ffa577fc90c4921"} err="failed to get container status \"49d0fc62354fd7ecf47bf54b59cf530f3b55936f6b95a5035ffa577fc90c4921\": rpc error: code = NotFound desc = could not find container \"49d0fc62354fd7ecf47bf54b59cf530f3b55936f6b95a5035ffa577fc90c4921\": container with ID starting with 49d0fc62354fd7ecf47bf54b59cf530f3b55936f6b95a5035ffa577fc90c4921 not found: ID does not exist" Oct 30 01:31:49.916565 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.916547 5300 scope.go:122] "RemoveContainer" containerID="1c809ade95391c4ccf74e39c84d8c0d7db863564a3e391a193368d41af1da592" Oct 30 01:31:49.927596 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.927568 5300 scope.go:122] "RemoveContainer" containerID="1c809ade95391c4ccf74e39c84d8c0d7db863564a3e391a193368d41af1da592" Oct 30 01:31:49.928130 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:31:49.928097 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c809ade95391c4ccf74e39c84d8c0d7db863564a3e391a193368d41af1da592\": container with ID starting with 1c809ade95391c4ccf74e39c84d8c0d7db863564a3e391a193368d41af1da592 not found: ID does not exist" containerID="1c809ade95391c4ccf74e39c84d8c0d7db863564a3e391a193368d41af1da592" Oct 30 01:31:49.928130 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.928128 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c809ade95391c4ccf74e39c84d8c0d7db863564a3e391a193368d41af1da592"} err="failed to get container status \"1c809ade95391c4ccf74e39c84d8c0d7db863564a3e391a193368d41af1da592\": rpc error: code = NotFound desc = could not find container \"1c809ade95391c4ccf74e39c84d8c0d7db863564a3e391a193368d41af1da592\": container with ID starting with 1c809ade95391c4ccf74e39c84d8c0d7db863564a3e391a193368d41af1da592 not found: ID does not exist" Oct 30 01:31:49.928418 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.928151 5300 scope.go:122] "RemoveContainer" containerID="c2b85899622be1e89574dfb021893a6f5c66d0fc7a6167fcd129ac09aa300a60" Oct 30 01:31:49.937381 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.937359 5300 scope.go:122] "RemoveContainer" containerID="c2b85899622be1e89574dfb021893a6f5c66d0fc7a6167fcd129ac09aa300a60" Oct 30 01:31:49.937651 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:31:49.937633 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2b85899622be1e89574dfb021893a6f5c66d0fc7a6167fcd129ac09aa300a60\": container with ID starting with c2b85899622be1e89574dfb021893a6f5c66d0fc7a6167fcd129ac09aa300a60 not found: ID does not exist" containerID="c2b85899622be1e89574dfb021893a6f5c66d0fc7a6167fcd129ac09aa300a60" Oct 30 01:31:49.937698 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.937657 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2b85899622be1e89574dfb021893a6f5c66d0fc7a6167fcd129ac09aa300a60"} err="failed to get container status \"c2b85899622be1e89574dfb021893a6f5c66d0fc7a6167fcd129ac09aa300a60\": rpc error: code = NotFound desc = could not find container \"c2b85899622be1e89574dfb021893a6f5c66d0fc7a6167fcd129ac09aa300a60\": container with ID starting with c2b85899622be1e89574dfb021893a6f5c66d0fc7a6167fcd129ac09aa300a60 not found: ID does not exist" Oct 30 01:31:49.978326 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.978297 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq"] Oct 30 01:31:49.994609 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:49.994578 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-cqzvq"] Oct 30 01:31:50.115821 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:50.115702 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl"] Oct 30 01:31:50.134080 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:50.134044 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-x5xnl"] Oct 30 01:31:50.173496 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:50.173466 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx"] Oct 30 01:31:50.176907 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:50.176882 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-886adb-openshift-default-64589cdbfc-r4nlx"] Oct 30 01:31:51.468829 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.468800 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:51.594003 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:31:51.593971 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:31:51.611150 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.611092 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:31:51.680209 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.680182 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-workload-certs\") pod \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " Oct 30 01:31:51.680697 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.680224 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-token\" (UniqueName: \"kubernetes.io/projected/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-token\") pod \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " Oct 30 01:31:51.680697 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.680248 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-envoy\") pod \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " Oct 30 01:31:51.680697 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.680287 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istiod-ca-cert\") pod \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " Oct 30 01:31:51.680697 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.680320 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-kube-api-access-s4nfg\" (UniqueName: \"kubernetes.io/projected/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-kube-api-access-s4nfg\") pod \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " Oct 30 01:31:51.680697 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.680355 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-data\") pod \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " Oct 30 01:31:51.680697 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.680382 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-credential-socket\") pod \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " Oct 30 01:31:51.680697 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.680468 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-workload-socket\") pod \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " Oct 30 01:31:51.680697 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.680538 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-podinfo\") pod \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\" (UID: \"f79971c3-5d7b-4fd1-a84f-46b2c0190af6\") " Oct 30 01:31:51.680697 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.680561 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-workload-certs" pod "f79971c3-5d7b-4fd1-a84f-46b2c0190af6" (UID: "f79971c3-5d7b-4fd1-a84f-46b2c0190af6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:51.681360 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.680755 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istiod-ca-cert" pod "f79971c3-5d7b-4fd1-a84f-46b2c0190af6" (UID: "f79971c3-5d7b-4fd1-a84f-46b2c0190af6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:31:51.681360 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.680971 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-data" pod "f79971c3-5d7b-4fd1-a84f-46b2c0190af6" (UID: "f79971c3-5d7b-4fd1-a84f-46b2c0190af6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:51.681360 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.681101 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-credential-socket" pod "f79971c3-5d7b-4fd1-a84f-46b2c0190af6" (UID: "f79971c3-5d7b-4fd1-a84f-46b2c0190af6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:51.681360 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.681171 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-workload-socket" pod "f79971c3-5d7b-4fd1-a84f-46b2c0190af6" (UID: "f79971c3-5d7b-4fd1-a84f-46b2c0190af6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:51.681360 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.681286 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:51.681360 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.681306 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:51.681360 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.681321 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:51.681360 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.681333 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:51.683284 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.683234 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-envoy" pod "f79971c3-5d7b-4fd1-a84f-46b2c0190af6" (UID: "f79971c3-5d7b-4fd1-a84f-46b2c0190af6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:51.683614 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.683585 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-kube-api-access-s4nfg" pod "f79971c3-5d7b-4fd1-a84f-46b2c0190af6" (UID: "f79971c3-5d7b-4fd1-a84f-46b2c0190af6"). InnerVolumeSpecName "kube-api-access-s4nfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:51.684262 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.684112 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-token" pod "f79971c3-5d7b-4fd1-a84f-46b2c0190af6" (UID: "f79971c3-5d7b-4fd1-a84f-46b2c0190af6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:51.684262 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.684217 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-podinfo" pod "f79971c3-5d7b-4fd1-a84f-46b2c0190af6" (UID: "f79971c3-5d7b-4fd1-a84f-46b2c0190af6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:31:51.782433 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.782407 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:51.782433 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.782430 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:51.782615 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.782441 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4nfg\" (UniqueName: \"kubernetes.io/projected/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-kube-api-access-s4nfg\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:51.782615 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.782452 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:51.782615 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.782461 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f79971c3-5d7b-4fd1-a84f-46b2c0190af6-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:51.820387 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.820347 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24445fd3-1ef0-4523-9548-7d1400bcb32b" path="/var/lib/kubelet/pods/24445fd3-1ef0-4523-9548-7d1400bcb32b/volumes" Oct 30 01:31:51.820940 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.820921 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="508889de-4033-41dd-9b1a-88925f799310" path="/var/lib/kubelet/pods/508889de-4033-41dd-9b1a-88925f799310/volumes" Oct 30 01:31:51.821428 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.821410 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfadd1c5-3001-4ebb-8156-46628aa2e722" path="/var/lib/kubelet/pods/cfadd1c5-3001-4ebb-8156-46628aa2e722/volumes" Oct 30 01:31:51.924169 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.924091 5300 generic.go:356] "Generic (PLEG): container finished" podID="f79971c3-5d7b-4fd1-a84f-46b2c0190af6" containerID="1a5d65f474e166141d53f1499262266ca0ca1fddece615a60b1b2b9a8a1c8945" exitCode=0 Oct 30 01:31:51.924169 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.924136 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" Oct 30 01:31:51.924348 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.924169 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" event={"ID":"f79971c3-5d7b-4fd1-a84f-46b2c0190af6","Type":"ContainerDied","Data":"1a5d65f474e166141d53f1499262266ca0ca1fddece615a60b1b2b9a8a1c8945"} Oct 30 01:31:51.924348 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.924220 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6" event={"ID":"f79971c3-5d7b-4fd1-a84f-46b2c0190af6","Type":"ContainerDied","Data":"9717da3ccbd6e555b97ba0c27e0f476a922dba63a7d82a41e285a042d0fbae4d"} Oct 30 01:31:51.924348 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.924242 5300 scope.go:122] "RemoveContainer" containerID="1a5d65f474e166141d53f1499262266ca0ca1fddece615a60b1b2b9a8a1c8945" Oct 30 01:31:51.933790 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.933744 5300 scope.go:122] "RemoveContainer" containerID="1a5d65f474e166141d53f1499262266ca0ca1fddece615a60b1b2b9a8a1c8945" Oct 30 01:31:51.934160 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:31:51.934134 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a5d65f474e166141d53f1499262266ca0ca1fddece615a60b1b2b9a8a1c8945\": container with ID starting with 1a5d65f474e166141d53f1499262266ca0ca1fddece615a60b1b2b9a8a1c8945 not found: ID does not exist" containerID="1a5d65f474e166141d53f1499262266ca0ca1fddece615a60b1b2b9a8a1c8945" Oct 30 01:31:51.934225 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.934170 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a5d65f474e166141d53f1499262266ca0ca1fddece615a60b1b2b9a8a1c8945"} err="failed to get container status \"1a5d65f474e166141d53f1499262266ca0ca1fddece615a60b1b2b9a8a1c8945\": rpc error: code = NotFound desc = could not find container \"1a5d65f474e166141d53f1499262266ca0ca1fddece615a60b1b2b9a8a1c8945\": container with ID starting with 1a5d65f474e166141d53f1499262266ca0ca1fddece615a60b1b2b9a8a1c8945 not found: ID does not exist" Oct 30 01:31:51.982140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.982112 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6"] Oct 30 01:31:51.988159 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:51.988139 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-d81f0a-openshift-default-8499d4d47f-6kmt6"] Oct 30 01:31:53.131025 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:53.130980 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:53.820284 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:53.820246 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f79971c3-5d7b-4fd1-a84f-46b2c0190af6" path="/var/lib/kubelet/pods/f79971c3-5d7b-4fd1-a84f-46b2c0190af6/volumes" Oct 30 01:31:54.595112 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:54.595077 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:55.002204 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:55.002163 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:55.100900 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:55.100845 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:56.419136 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:56.419095 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:57.269452 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:57.269412 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh"] Oct 30 01:31:57.303485 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:57.303450 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt"] Oct 30 01:31:57.418889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:57.418860 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:57.517051 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:57.517015 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:58.002999 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:58.002957 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:59.346829 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:59.346786 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw"] Oct 30 01:31:59.347348 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:59.347020 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" podUID="016eaf4d-095e-4eec-9cae-2c3cb12e13b6" containerName="maas-api" containerID="cri-o://2c6ac8c3ed1c9398988a5417c7bbe9a8a323bde6bd3d14f2faed04fa5c7cfd76" gracePeriod=30 Oct 30 01:31:59.387291 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:59.387260 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:59.647099 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:59.647068 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" Oct 30 01:31:59.768779 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:59.768730 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-maas-api-tls\") pod \"016eaf4d-095e-4eec-9cae-2c3cb12e13b6\" (UID: \"016eaf4d-095e-4eec-9cae-2c3cb12e13b6\") " Oct 30 01:31:59.768975 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:59.768873 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-openshift-service-ca\") pod \"016eaf4d-095e-4eec-9cae-2c3cb12e13b6\" (UID: \"016eaf4d-095e-4eec-9cae-2c3cb12e13b6\") " Oct 30 01:31:59.768975 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:59.768907 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-metrics-certs\") pod \"016eaf4d-095e-4eec-9cae-2c3cb12e13b6\" (UID: \"016eaf4d-095e-4eec-9cae-2c3cb12e13b6\") " Oct 30 01:31:59.768975 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:59.768964 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-kube-api-access-bhjkb\" (UniqueName: \"kubernetes.io/projected/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-kube-api-access-bhjkb\") pod \"016eaf4d-095e-4eec-9cae-2c3cb12e13b6\" (UID: \"016eaf4d-095e-4eec-9cae-2c3cb12e13b6\") " Oct 30 01:31:59.769300 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:59.769267 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-openshift-service-ca" pod "016eaf4d-095e-4eec-9cae-2c3cb12e13b6" (UID: "016eaf4d-095e-4eec-9cae-2c3cb12e13b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:31:59.770953 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:59.770932 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-maas-api-tls" pod "016eaf4d-095e-4eec-9cae-2c3cb12e13b6" (UID: "016eaf4d-095e-4eec-9cae-2c3cb12e13b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:31:59.771119 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:59.771098 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-kube-api-access-bhjkb" pod "016eaf4d-095e-4eec-9cae-2c3cb12e13b6" (UID: "016eaf4d-095e-4eec-9cae-2c3cb12e13b6"). InnerVolumeSpecName "kube-api-access-bhjkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:59.771186 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:59.771167 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-metrics-certs" pod "016eaf4d-095e-4eec-9cae-2c3cb12e13b6" (UID: "016eaf4d-095e-4eec-9cae-2c3cb12e13b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:31:59.862718 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:59.862684 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:59.870068 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:59.870034 5300 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-openshift-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:59.870068 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:59.870064 5300 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-metrics-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:59.870068 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:59.870077 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhjkb\" (UniqueName: \"kubernetes.io/projected/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-kube-api-access-bhjkb\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:59.870314 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:59.870085 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016eaf4d-095e-4eec-9cae-2c3cb12e13b6-maas-api-tls\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:31:59.961991 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:59.961908 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:59.969554 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:59.969500 5300 generic.go:356] "Generic (PLEG): container finished" podID="016eaf4d-095e-4eec-9cae-2c3cb12e13b6" containerID="2c6ac8c3ed1c9398988a5417c7bbe9a8a323bde6bd3d14f2faed04fa5c7cfd76" exitCode=0 Oct 30 01:31:59.969747 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:59.969615 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" event={"ID":"016eaf4d-095e-4eec-9cae-2c3cb12e13b6","Type":"ContainerDied","Data":"2c6ac8c3ed1c9398988a5417c7bbe9a8a323bde6bd3d14f2faed04fa5c7cfd76"} Oct 30 01:31:59.969747 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:59.969642 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" Oct 30 01:31:59.969747 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:59.969666 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw" event={"ID":"016eaf4d-095e-4eec-9cae-2c3cb12e13b6","Type":"ContainerDied","Data":"0408076955c857e5da697777a13cd87e294a11f680e7b9b856bbfd5f253ea7c6"} Oct 30 01:31:59.969747 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:59.969692 5300 scope.go:122] "RemoveContainer" containerID="2c6ac8c3ed1c9398988a5417c7bbe9a8a323bde6bd3d14f2faed04fa5c7cfd76" Oct 30 01:31:59.981561 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:59.981536 5300 scope.go:122] "RemoveContainer" containerID="2c6ac8c3ed1c9398988a5417c7bbe9a8a323bde6bd3d14f2faed04fa5c7cfd76" Oct 30 01:31:59.981917 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:31:59.981888 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c6ac8c3ed1c9398988a5417c7bbe9a8a323bde6bd3d14f2faed04fa5c7cfd76\": container with ID starting with 2c6ac8c3ed1c9398988a5417c7bbe9a8a323bde6bd3d14f2faed04fa5c7cfd76 not found: ID does not exist" containerID="2c6ac8c3ed1c9398988a5417c7bbe9a8a323bde6bd3d14f2faed04fa5c7cfd76" Oct 30 01:31:59.981998 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:31:59.981931 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c6ac8c3ed1c9398988a5417c7bbe9a8a323bde6bd3d14f2faed04fa5c7cfd76"} err="failed to get container status \"2c6ac8c3ed1c9398988a5417c7bbe9a8a323bde6bd3d14f2faed04fa5c7cfd76\": rpc error: code = NotFound desc = could not find container \"2c6ac8c3ed1c9398988a5417c7bbe9a8a323bde6bd3d14f2faed04fa5c7cfd76\": container with ID starting with 2c6ac8c3ed1c9398988a5417c7bbe9a8a323bde6bd3d14f2faed04fa5c7cfd76 not found: ID does not exist" Oct 30 01:32:00.032296 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:00.032252 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw"] Oct 30 01:32:00.038281 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:00.038244 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-e92648-848bbd7c56-b8rzw"] Oct 30 01:32:00.080101 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:00.080067 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:00.484838 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:00.484805 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:01.822058 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:01.822015 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="016eaf4d-095e-4eec-9cae-2c3cb12e13b6" path="/var/lib/kubelet/pods/016eaf4d-095e-4eec-9cae-2c3cb12e13b6/volumes" Oct 30 01:32:02.076830 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:02.076724 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:02.270111 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:02.270058 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh" podUID="aa7f5966-fe14-4c2c-af88-ac82100209ba" containerName="payload-processing" containerID="cri-o://0cd561fc955ec100b8d6ddcd2a858e9c4c5004e428c9235caa8e915b013486c7" gracePeriod=25 Oct 30 01:32:02.304223 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:02.304173 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt" podUID="19df0d00-62d5-4c6c-9cba-673cd02dbb8a" containerName="payload-pre-processing" containerID="cri-o://3d1bc8568f9354600975d6e738079c2dfba0e59d31b12455e7ed2b14bd1e2b6c" gracePeriod=25 Oct 30 01:32:03.983650 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:03.983610 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt" podUID="19df0d00-62d5-4c6c-9cba-673cd02dbb8a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:32:03.984190 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:03.983647 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh" podUID="aa7f5966-fe14-4c2c-af88-ac82100209ba" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:32:04.103473 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:04.103437 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:04.582294 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:04.582234 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" containerID="cri-o://9f42a87c220017a47e98887cd5d046a55c03878453153acf49414a9ea13a7b04" gracePeriod=45 Oct 30 01:32:04.632816 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:04.632749 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": dial tcp 10.133.0.60:8000: connect: connection refused" Oct 30 01:32:04.856556 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:04.856531 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 01:32:04.928737 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:04.928694 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-model-cache\") pod \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\" (UID: \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\") " Oct 30 01:32:04.928964 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:04.928830 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-home\" (UniqueName: \"kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-home\") pod \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\" (UID: \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\") " Oct 30 01:32:04.928964 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:04.928947 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-dshm\") pod \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\" (UID: \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\") " Oct 30 01:32:04.929095 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:04.928996 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da1ea263-175e-4b07-b74d-a30b16d9ed8c-tls-certs\" (UniqueName: \"kubernetes.io/secret/da1ea263-175e-4b07-b74d-a30b16d9ed8c-tls-certs\") pod \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\" (UID: \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\") " Oct 30 01:32:04.929095 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:04.929036 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da1ea263-175e-4b07-b74d-a30b16d9ed8c-kube-api-access-4n9xc\" (UniqueName: \"kubernetes.io/projected/da1ea263-175e-4b07-b74d-a30b16d9ed8c-kube-api-access-4n9xc\") pod \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\" (UID: \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\") " Oct 30 01:32:04.929095 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:04.929028 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-model-cache" pod "da1ea263-175e-4b07-b74d-a30b16d9ed8c" (UID: "da1ea263-175e-4b07-b74d-a30b16d9ed8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:04.929095 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:04.929064 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-tmp-dir\") pod \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\" (UID: \"da1ea263-175e-4b07-b74d-a30b16d9ed8c\") " Oct 30 01:32:04.929298 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:04.929097 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-home" pod "da1ea263-175e-4b07-b74d-a30b16d9ed8c" (UID: "da1ea263-175e-4b07-b74d-a30b16d9ed8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:04.929351 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:04.929302 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-tmp-dir" pod "da1ea263-175e-4b07-b74d-a30b16d9ed8c" (UID: "da1ea263-175e-4b07-b74d-a30b16d9ed8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:04.929625 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:04.929602 5300 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-tmp-dir\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:04.930059 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:04.930041 5300 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-model-cache\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:04.930215 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:04.930063 5300 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-home\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:04.931562 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:04.931536 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1ea263-175e-4b07-b74d-a30b16d9ed8c-kube-api-access-4n9xc" pod "da1ea263-175e-4b07-b74d-a30b16d9ed8c" (UID: "da1ea263-175e-4b07-b74d-a30b16d9ed8c"). InnerVolumeSpecName "kube-api-access-4n9xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:04.932210 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:04.932168 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da1ea263-175e-4b07-b74d-a30b16d9ed8c-tls-certs" pod "da1ea263-175e-4b07-b74d-a30b16d9ed8c" (UID: "da1ea263-175e-4b07-b74d-a30b16d9ed8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:04.932312 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:04.932207 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-dshm" pod "da1ea263-175e-4b07-b74d-a30b16d9ed8c" (UID: "da1ea263-175e-4b07-b74d-a30b16d9ed8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:04.999605 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:04.999571 5300 generic.go:356] "Generic (PLEG): container finished" podID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerID="9f42a87c220017a47e98887cd5d046a55c03878453153acf49414a9ea13a7b04" exitCode=0 Oct 30 01:32:05.000076 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:04.999613 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" Oct 30 01:32:05.000076 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:04.999653 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" event={"ID":"da1ea263-175e-4b07-b74d-a30b16d9ed8c","Type":"ContainerDied","Data":"9f42a87c220017a47e98887cd5d046a55c03878453153acf49414a9ea13a7b04"} Oct 30 01:32:05.000076 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:04.999696 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms" event={"ID":"da1ea263-175e-4b07-b74d-a30b16d9ed8c","Type":"ContainerDied","Data":"93f7a209a6eac153b0749608017bab4ccbd94abc1d46cf89e3370556f9ede1e5"} Oct 30 01:32:05.000076 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:04.999713 5300 scope.go:122] "RemoveContainer" containerID="9f42a87c220017a47e98887cd5d046a55c03878453153acf49414a9ea13a7b04" Oct 30 01:32:05.010843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:05.010806 5300 scope.go:122] "RemoveContainer" containerID="9f42a87c220017a47e98887cd5d046a55c03878453153acf49414a9ea13a7b04" Oct 30 01:32:05.011438 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:32:05.011388 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f42a87c220017a47e98887cd5d046a55c03878453153acf49414a9ea13a7b04\": container with ID starting with 9f42a87c220017a47e98887cd5d046a55c03878453153acf49414a9ea13a7b04 not found: ID does not exist" containerID="9f42a87c220017a47e98887cd5d046a55c03878453153acf49414a9ea13a7b04" Oct 30 01:32:05.011545 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:05.011437 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f42a87c220017a47e98887cd5d046a55c03878453153acf49414a9ea13a7b04"} err="failed to get container status \"9f42a87c220017a47e98887cd5d046a55c03878453153acf49414a9ea13a7b04\": rpc error: code = NotFound desc = could not find container \"9f42a87c220017a47e98887cd5d046a55c03878453153acf49414a9ea13a7b04\": container with ID starting with 9f42a87c220017a47e98887cd5d046a55c03878453153acf49414a9ea13a7b04 not found: ID does not exist" Oct 30 01:32:05.031502 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:05.031473 5300 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da1ea263-175e-4b07-b74d-a30b16d9ed8c-dshm\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:05.031502 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:05.031502 5300 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da1ea263-175e-4b07-b74d-a30b16d9ed8c-tls-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:05.031502 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:05.031514 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n9xc\" (UniqueName: \"kubernetes.io/projected/da1ea263-175e-4b07-b74d-a30b16d9ed8c-kube-api-access-4n9xc\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:05.057144 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:05.057109 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms"] Oct 30 01:32:05.060825 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:05.060796 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-82b647/premium-w3-82b647-kserve-c9979684b-j6hms"] Oct 30 01:32:05.590731 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:05.590694 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:05.820513 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:05.820477 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" path="/var/lib/kubelet/pods/da1ea263-175e-4b07-b74d-a30b16d9ed8c/volumes" Oct 30 01:32:06.883478 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:06.883448 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:07.011398 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:07.011357 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:07.210140 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:07.210038 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb"] Oct 30 01:32:07.210320 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:07.210296 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" podUID="27a25c57-e305-43ad-9de6-e32231466d85" containerName="istio-proxy" containerID="cri-o://9940ffc6cb6a1001f4be5808d6862b58dcdfa3ebc2bebeafed145eb02c6c7f58" gracePeriod=30 Oct 30 01:32:07.223444 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:07.223415 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8"] Oct 30 01:32:07.223644 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:07.223624 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" podUID="459d912e-ed51-4c50-a747-3a2767b30ee2" containerName="istio-proxy" containerID="cri-o://1f7bd68f02bd6653d44459059816ccff225f9957c56721ce68211ec0c76dadd9" gracePeriod=30 Oct 30 01:32:07.228774 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:07.228729 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6"] Oct 30 01:32:07.229010 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:07.228978 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" podUID="1e2be936-3c18-4d9b-87a4-800cb91748b7" containerName="istio-proxy" containerID="cri-o://56ec838d06e5d7f9ef6b0da56927c45ef9cdf63ae351ac3ff01c011132cef27f" gracePeriod=30 Oct 30 01:32:07.239951 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:07.239921 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:07.389688 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:07.389655 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:07.680485 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:07.680455 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:09.191410 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:09.191377 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:10.801288 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:10.801258 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:12.351283 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.351248 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:12.504526 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.504497 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:32:12.514955 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.514930 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:32:12.551743 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.551715 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:32:12.612781 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.612725 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-podinfo\") pod \"459d912e-ed51-4c50-a747-3a2767b30ee2\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " Oct 30 01:32:12.612985 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.612806 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-istio-envoy\") pod \"27a25c57-e305-43ad-9de6-e32231466d85\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " Oct 30 01:32:12.612985 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.612830 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-token\" (UniqueName: \"kubernetes.io/projected/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-token\") pod \"459d912e-ed51-4c50-a747-3a2767b30ee2\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " Oct 30 01:32:12.612985 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.612849 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/459d912e-ed51-4c50-a747-3a2767b30ee2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/459d912e-ed51-4c50-a747-3a2767b30ee2-istiod-ca-cert\") pod \"459d912e-ed51-4c50-a747-3a2767b30ee2\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " Oct 30 01:32:12.612985 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.612876 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-credential-socket\") pod \"27a25c57-e305-43ad-9de6-e32231466d85\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " Oct 30 01:32:12.612985 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.612902 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-workload-socket\") pod \"27a25c57-e305-43ad-9de6-e32231466d85\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " Oct 30 01:32:12.613286 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.613069 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-credential-socket\") pod \"459d912e-ed51-4c50-a747-3a2767b30ee2\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " Oct 30 01:32:12.613286 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.613127 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-workload-certs\") pod \"459d912e-ed51-4c50-a747-3a2767b30ee2\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " Oct 30 01:32:12.613286 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.613173 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/459d912e-ed51-4c50-a747-3a2767b30ee2-kube-api-access-fw4qd\" (UniqueName: \"kubernetes.io/projected/459d912e-ed51-4c50-a747-3a2767b30ee2-kube-api-access-fw4qd\") pod \"459d912e-ed51-4c50-a747-3a2767b30ee2\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " Oct 30 01:32:12.613286 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.613250 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-istio-data\") pod \"27a25c57-e305-43ad-9de6-e32231466d85\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " Oct 30 01:32:12.613286 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.613172 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-credential-socket" pod "27a25c57-e305-43ad-9de6-e32231466d85" (UID: "27a25c57-e305-43ad-9de6-e32231466d85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:12.613556 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.613286 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/459d912e-ed51-4c50-a747-3a2767b30ee2-istiod-ca-cert" pod "459d912e-ed51-4c50-a747-3a2767b30ee2" (UID: "459d912e-ed51-4c50-a747-3a2767b30ee2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:12.613556 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.613292 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-workload-socket\") pod \"459d912e-ed51-4c50-a747-3a2767b30ee2\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " Oct 30 01:32:12.613556 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.613233 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-workload-socket" pod "27a25c57-e305-43ad-9de6-e32231466d85" (UID: "27a25c57-e305-43ad-9de6-e32231466d85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:12.613556 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.613366 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-data\") pod \"459d912e-ed51-4c50-a747-3a2767b30ee2\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " Oct 30 01:32:12.613556 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.613392 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-credential-socket" pod "459d912e-ed51-4c50-a747-3a2767b30ee2" (UID: "459d912e-ed51-4c50-a747-3a2767b30ee2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:12.613556 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.613404 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27a25c57-e305-43ad-9de6-e32231466d85-kube-api-access-nrcsk\" (UniqueName: \"kubernetes.io/projected/27a25c57-e305-43ad-9de6-e32231466d85-kube-api-access-nrcsk\") pod \"27a25c57-e305-43ad-9de6-e32231466d85\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " Oct 30 01:32:12.613556 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.613467 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27a25c57-e305-43ad-9de6-e32231466d85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27a25c57-e305-43ad-9de6-e32231466d85-istio-podinfo\") pod \"27a25c57-e305-43ad-9de6-e32231466d85\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " Oct 30 01:32:12.613556 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.613508 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27a25c57-e305-43ad-9de6-e32231466d85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27a25c57-e305-43ad-9de6-e32231466d85-istiod-ca-cert\") pod \"27a25c57-e305-43ad-9de6-e32231466d85\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " Oct 30 01:32:12.613556 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.613544 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-envoy\") pod \"459d912e-ed51-4c50-a747-3a2767b30ee2\" (UID: \"459d912e-ed51-4c50-a747-3a2767b30ee2\") " Oct 30 01:32:12.614088 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.613579 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-workload-certs\") pod \"27a25c57-e305-43ad-9de6-e32231466d85\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " Oct 30 01:32:12.614088 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.613604 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-workload-certs" pod "459d912e-ed51-4c50-a747-3a2767b30ee2" (UID: "459d912e-ed51-4c50-a747-3a2767b30ee2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:12.614088 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.613635 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27a25c57-e305-43ad-9de6-e32231466d85-istio-token\" (UniqueName: \"kubernetes.io/projected/27a25c57-e305-43ad-9de6-e32231466d85-istio-token\") pod \"27a25c57-e305-43ad-9de6-e32231466d85\" (UID: \"27a25c57-e305-43ad-9de6-e32231466d85\") " Oct 30 01:32:12.614088 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.613643 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-workload-socket" pod "459d912e-ed51-4c50-a747-3a2767b30ee2" (UID: "459d912e-ed51-4c50-a747-3a2767b30ee2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:12.614088 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.613656 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-data" pod "459d912e-ed51-4c50-a747-3a2767b30ee2" (UID: "459d912e-ed51-4c50-a747-3a2767b30ee2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:12.614088 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.613960 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27a25c57-e305-43ad-9de6-e32231466d85-istiod-ca-cert" pod "27a25c57-e305-43ad-9de6-e32231466d85" (UID: "27a25c57-e305-43ad-9de6-e32231466d85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:12.614088 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.614011 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-istio-data" pod "27a25c57-e305-43ad-9de6-e32231466d85" (UID: "27a25c57-e305-43ad-9de6-e32231466d85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:12.614507 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.614177 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.614507 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.614199 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.614507 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.614213 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.614507 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.614228 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.614507 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.614244 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27a25c57-e305-43ad-9de6-e32231466d85-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.614507 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.614255 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/459d912e-ed51-4c50-a747-3a2767b30ee2-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.614507 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.614268 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.614507 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.614283 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.614507 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.614293 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-workload-certs" pod "27a25c57-e305-43ad-9de6-e32231466d85" (UID: "27a25c57-e305-43ad-9de6-e32231466d85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:12.614507 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.614302 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.615880 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.615721 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-podinfo" pod "459d912e-ed51-4c50-a747-3a2767b30ee2" (UID: "459d912e-ed51-4c50-a747-3a2767b30ee2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:32:12.615880 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.615797 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-token" pod "459d912e-ed51-4c50-a747-3a2767b30ee2" (UID: "459d912e-ed51-4c50-a747-3a2767b30ee2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:12.615880 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.615843 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459d912e-ed51-4c50-a747-3a2767b30ee2-kube-api-access-fw4qd" pod "459d912e-ed51-4c50-a747-3a2767b30ee2" (UID: "459d912e-ed51-4c50-a747-3a2767b30ee2"). InnerVolumeSpecName "kube-api-access-fw4qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:12.616476 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.616452 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-envoy" pod "459d912e-ed51-4c50-a747-3a2767b30ee2" (UID: "459d912e-ed51-4c50-a747-3a2767b30ee2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:12.616476 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.616465 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a25c57-e305-43ad-9de6-e32231466d85-kube-api-access-nrcsk" pod "27a25c57-e305-43ad-9de6-e32231466d85" (UID: "27a25c57-e305-43ad-9de6-e32231466d85"). InnerVolumeSpecName "kube-api-access-nrcsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:12.616659 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.616512 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a25c57-e305-43ad-9de6-e32231466d85-istio-token" pod "27a25c57-e305-43ad-9de6-e32231466d85" (UID: "27a25c57-e305-43ad-9de6-e32231466d85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:12.616750 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.616732 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-istio-envoy" pod "27a25c57-e305-43ad-9de6-e32231466d85" (UID: "27a25c57-e305-43ad-9de6-e32231466d85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:12.617211 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.617191 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27a25c57-e305-43ad-9de6-e32231466d85-istio-podinfo" pod "27a25c57-e305-43ad-9de6-e32231466d85" (UID: "27a25c57-e305-43ad-9de6-e32231466d85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:32:12.658637 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.658605 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4"] Oct 30 01:32:12.692839 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.692808 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct"] Oct 30 01:32:12.714595 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.714565 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-credential-socket\") pod \"1e2be936-3c18-4d9b-87a4-800cb91748b7\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " Oct 30 01:32:12.714789 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.714618 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-workload-socket\") pod \"1e2be936-3c18-4d9b-87a4-800cb91748b7\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " Oct 30 01:32:12.714789 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.714700 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-envoy\") pod \"1e2be936-3c18-4d9b-87a4-800cb91748b7\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " Oct 30 01:32:12.714789 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.714751 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-podinfo\") pod \"1e2be936-3c18-4d9b-87a4-800cb91748b7\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " Oct 30 01:32:12.714999 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.714791 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-workload-certs\") pod \"1e2be936-3c18-4d9b-87a4-800cb91748b7\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " Oct 30 01:32:12.714999 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.714838 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-data\") pod \"1e2be936-3c18-4d9b-87a4-800cb91748b7\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " Oct 30 01:32:12.714999 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.714860 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-credential-socket" pod "1e2be936-3c18-4d9b-87a4-800cb91748b7" (UID: "1e2be936-3c18-4d9b-87a4-800cb91748b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:12.714999 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.714870 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-token\" (UniqueName: \"kubernetes.io/projected/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-token\") pod \"1e2be936-3c18-4d9b-87a4-800cb91748b7\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " Oct 30 01:32:12.714999 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.714927 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e2be936-3c18-4d9b-87a4-800cb91748b7-kube-api-access-7k2lq\" (UniqueName: \"kubernetes.io/projected/1e2be936-3c18-4d9b-87a4-800cb91748b7-kube-api-access-7k2lq\") pod \"1e2be936-3c18-4d9b-87a4-800cb91748b7\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " Oct 30 01:32:12.714999 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.714969 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e2be936-3c18-4d9b-87a4-800cb91748b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e2be936-3c18-4d9b-87a4-800cb91748b7-istiod-ca-cert\") pod \"1e2be936-3c18-4d9b-87a4-800cb91748b7\" (UID: \"1e2be936-3c18-4d9b-87a4-800cb91748b7\") " Oct 30 01:32:12.715297 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.715084 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-workload-certs" pod "1e2be936-3c18-4d9b-87a4-800cb91748b7" (UID: "1e2be936-3c18-4d9b-87a4-800cb91748b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:12.715420 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.715386 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.715420 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.715409 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.715933 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.715424 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.715933 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.715438 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw4qd\" (UniqueName: \"kubernetes.io/projected/459d912e-ed51-4c50-a747-3a2767b30ee2-kube-api-access-fw4qd\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.715933 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.715453 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.715933 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.715466 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrcsk\" (UniqueName: \"kubernetes.io/projected/27a25c57-e305-43ad-9de6-e32231466d85-kube-api-access-nrcsk\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.715933 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.715478 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27a25c57-e305-43ad-9de6-e32231466d85-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.715933 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.715490 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/459d912e-ed51-4c50-a747-3a2767b30ee2-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.715933 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.715501 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.715933 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.715512 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27a25c57-e305-43ad-9de6-e32231466d85-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.715933 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.715524 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27a25c57-e305-43ad-9de6-e32231466d85-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.715933 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.715532 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-data" pod "1e2be936-3c18-4d9b-87a4-800cb91748b7" (UID: "1e2be936-3c18-4d9b-87a4-800cb91748b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:12.715933 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.715004 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-workload-socket" pod "1e2be936-3c18-4d9b-87a4-800cb91748b7" (UID: "1e2be936-3c18-4d9b-87a4-800cb91748b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:12.715933 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.715838 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e2be936-3c18-4d9b-87a4-800cb91748b7-istiod-ca-cert" pod "1e2be936-3c18-4d9b-87a4-800cb91748b7" (UID: "1e2be936-3c18-4d9b-87a4-800cb91748b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:12.717254 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.717227 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-podinfo" pod "1e2be936-3c18-4d9b-87a4-800cb91748b7" (UID: "1e2be936-3c18-4d9b-87a4-800cb91748b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:32:12.718527 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.718250 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-token" pod "1e2be936-3c18-4d9b-87a4-800cb91748b7" (UID: "1e2be936-3c18-4d9b-87a4-800cb91748b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:12.718750 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.718723 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e2be936-3c18-4d9b-87a4-800cb91748b7-kube-api-access-7k2lq" pod "1e2be936-3c18-4d9b-87a4-800cb91748b7" (UID: "1e2be936-3c18-4d9b-87a4-800cb91748b7"). InnerVolumeSpecName "kube-api-access-7k2lq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:12.719549 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.719522 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-envoy" pod "1e2be936-3c18-4d9b-87a4-800cb91748b7" (UID: "1e2be936-3c18-4d9b-87a4-800cb91748b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:12.800151 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.800072 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:12.816935 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.816907 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.816935 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.816933 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.817137 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.816946 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.817137 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.816959 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.817137 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.816972 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e2be936-3c18-4d9b-87a4-800cb91748b7-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.817137 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.816985 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k2lq\" (UniqueName: \"kubernetes.io/projected/1e2be936-3c18-4d9b-87a4-800cb91748b7-kube-api-access-7k2lq\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:12.817137 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:12.816997 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e2be936-3c18-4d9b-87a4-800cb91748b7-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:13.051736 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.051638 5300 generic.go:356] "Generic (PLEG): container finished" podID="1e2be936-3c18-4d9b-87a4-800cb91748b7" containerID="56ec838d06e5d7f9ef6b0da56927c45ef9cdf63ae351ac3ff01c011132cef27f" exitCode=0 Oct 30 01:32:13.051736 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.051696 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" Oct 30 01:32:13.051736 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.051714 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" event={"ID":"1e2be936-3c18-4d9b-87a4-800cb91748b7","Type":"ContainerDied","Data":"56ec838d06e5d7f9ef6b0da56927c45ef9cdf63ae351ac3ff01c011132cef27f"} Oct 30 01:32:13.052054 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.051781 5300 scope.go:122] "RemoveContainer" containerID="56ec838d06e5d7f9ef6b0da56927c45ef9cdf63ae351ac3ff01c011132cef27f" Oct 30 01:32:13.052054 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.051793 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6" event={"ID":"1e2be936-3c18-4d9b-87a4-800cb91748b7","Type":"ContainerDied","Data":"61d4fb29919b310b3022182d1dab831385a71a2fa8c6d11b596c74218d5dcede"} Oct 30 01:32:13.053155 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.053135 5300 generic.go:356] "Generic (PLEG): container finished" podID="459d912e-ed51-4c50-a747-3a2767b30ee2" containerID="1f7bd68f02bd6653d44459059816ccff225f9957c56721ce68211ec0c76dadd9" exitCode=0 Oct 30 01:32:13.053282 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.053190 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" Oct 30 01:32:13.053282 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.053222 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" event={"ID":"459d912e-ed51-4c50-a747-3a2767b30ee2","Type":"ContainerDied","Data":"1f7bd68f02bd6653d44459059816ccff225f9957c56721ce68211ec0c76dadd9"} Oct 30 01:32:13.053282 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.053273 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8" event={"ID":"459d912e-ed51-4c50-a747-3a2767b30ee2","Type":"ContainerDied","Data":"cdc6bc54c979af78e4751fab521740c07a670e3e039df15ab6d8c20ae511fb62"} Oct 30 01:32:13.054726 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.054699 5300 generic.go:356] "Generic (PLEG): container finished" podID="27a25c57-e305-43ad-9de6-e32231466d85" containerID="9940ffc6cb6a1001f4be5808d6862b58dcdfa3ebc2bebeafed145eb02c6c7f58" exitCode=0 Oct 30 01:32:13.054877 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.054789 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" Oct 30 01:32:13.054877 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.054799 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" event={"ID":"27a25c57-e305-43ad-9de6-e32231466d85","Type":"ContainerDied","Data":"9940ffc6cb6a1001f4be5808d6862b58dcdfa3ebc2bebeafed145eb02c6c7f58"} Oct 30 01:32:13.054877 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.054843 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb" event={"ID":"27a25c57-e305-43ad-9de6-e32231466d85","Type":"ContainerDied","Data":"20f7d9154d2b3588584cb193103a06151913d603185dcf19f4adb4868158f7d9"} Oct 30 01:32:13.062888 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.062865 5300 scope.go:122] "RemoveContainer" containerID="56ec838d06e5d7f9ef6b0da56927c45ef9cdf63ae351ac3ff01c011132cef27f" Oct 30 01:32:13.063233 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:32:13.063209 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ec838d06e5d7f9ef6b0da56927c45ef9cdf63ae351ac3ff01c011132cef27f\": container with ID starting with 56ec838d06e5d7f9ef6b0da56927c45ef9cdf63ae351ac3ff01c011132cef27f not found: ID does not exist" containerID="56ec838d06e5d7f9ef6b0da56927c45ef9cdf63ae351ac3ff01c011132cef27f" Oct 30 01:32:13.063305 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.063244 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ec838d06e5d7f9ef6b0da56927c45ef9cdf63ae351ac3ff01c011132cef27f"} err="failed to get container status \"56ec838d06e5d7f9ef6b0da56927c45ef9cdf63ae351ac3ff01c011132cef27f\": rpc error: code = NotFound desc = could not find container \"56ec838d06e5d7f9ef6b0da56927c45ef9cdf63ae351ac3ff01c011132cef27f\": container with ID starting with 56ec838d06e5d7f9ef6b0da56927c45ef9cdf63ae351ac3ff01c011132cef27f not found: ID does not exist" Oct 30 01:32:13.063305 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.063267 5300 scope.go:122] "RemoveContainer" containerID="1f7bd68f02bd6653d44459059816ccff225f9957c56721ce68211ec0c76dadd9" Oct 30 01:32:13.073312 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.073296 5300 scope.go:122] "RemoveContainer" containerID="1f7bd68f02bd6653d44459059816ccff225f9957c56721ce68211ec0c76dadd9" Oct 30 01:32:13.073583 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:32:13.073565 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f7bd68f02bd6653d44459059816ccff225f9957c56721ce68211ec0c76dadd9\": container with ID starting with 1f7bd68f02bd6653d44459059816ccff225f9957c56721ce68211ec0c76dadd9 not found: ID does not exist" containerID="1f7bd68f02bd6653d44459059816ccff225f9957c56721ce68211ec0c76dadd9" Oct 30 01:32:13.073639 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.073591 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f7bd68f02bd6653d44459059816ccff225f9957c56721ce68211ec0c76dadd9"} err="failed to get container status \"1f7bd68f02bd6653d44459059816ccff225f9957c56721ce68211ec0c76dadd9\": rpc error: code = NotFound desc = could not find container \"1f7bd68f02bd6653d44459059816ccff225f9957c56721ce68211ec0c76dadd9\": container with ID starting with 1f7bd68f02bd6653d44459059816ccff225f9957c56721ce68211ec0c76dadd9 not found: ID does not exist" Oct 30 01:32:13.073639 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.073607 5300 scope.go:122] "RemoveContainer" containerID="9940ffc6cb6a1001f4be5808d6862b58dcdfa3ebc2bebeafed145eb02c6c7f58" Oct 30 01:32:13.081752 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.081736 5300 scope.go:122] "RemoveContainer" containerID="9940ffc6cb6a1001f4be5808d6862b58dcdfa3ebc2bebeafed145eb02c6c7f58" Oct 30 01:32:13.082049 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:32:13.082028 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9940ffc6cb6a1001f4be5808d6862b58dcdfa3ebc2bebeafed145eb02c6c7f58\": container with ID starting with 9940ffc6cb6a1001f4be5808d6862b58dcdfa3ebc2bebeafed145eb02c6c7f58 not found: ID does not exist" containerID="9940ffc6cb6a1001f4be5808d6862b58dcdfa3ebc2bebeafed145eb02c6c7f58" Oct 30 01:32:13.082099 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.082055 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9940ffc6cb6a1001f4be5808d6862b58dcdfa3ebc2bebeafed145eb02c6c7f58"} err="failed to get container status \"9940ffc6cb6a1001f4be5808d6862b58dcdfa3ebc2bebeafed145eb02c6c7f58\": rpc error: code = NotFound desc = could not find container \"9940ffc6cb6a1001f4be5808d6862b58dcdfa3ebc2bebeafed145eb02c6c7f58\": container with ID starting with 9940ffc6cb6a1001f4be5808d6862b58dcdfa3ebc2bebeafed145eb02c6c7f58 not found: ID does not exist" Oct 30 01:32:13.114209 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.114183 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6"] Oct 30 01:32:13.131001 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.128724 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-r4pg6"] Oct 30 01:32:13.167423 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.167387 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb"] Oct 30 01:32:13.171784 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.171737 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-zm4kb"] Oct 30 01:32:13.228231 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.228198 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8"] Oct 30 01:32:13.231997 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.231968 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e92648-openshift-default-6c4f9f8d45-w2jv8"] Oct 30 01:32:13.820102 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.820064 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e2be936-3c18-4d9b-87a4-800cb91748b7" path="/var/lib/kubelet/pods/1e2be936-3c18-4d9b-87a4-800cb91748b7/volumes" Oct 30 01:32:13.820629 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.820613 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27a25c57-e305-43ad-9de6-e32231466d85" path="/var/lib/kubelet/pods/27a25c57-e305-43ad-9de6-e32231466d85/volumes" Oct 30 01:32:13.821100 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.821086 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="459d912e-ed51-4c50-a747-3a2767b30ee2" path="/var/lib/kubelet/pods/459d912e-ed51-4c50-a747-3a2767b30ee2/volumes" Oct 30 01:32:13.948395 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.948357 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:13.983575 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.983534 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh" podUID="aa7f5966-fe14-4c2c-af88-ac82100209ba" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:32:13.983740 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:13.983533 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt" podUID="19df0d00-62d5-4c6c-9cba-673cd02dbb8a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:32:15.509032 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:15.508996 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:16.786596 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:16.786540 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:17.222290 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:17.222252 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:17.321280 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:17.321245 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:17.429576 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:17.429540 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:17.840676 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:17.840639 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:18.281873 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:18.281840 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9"] Oct 30 01:32:18.420286 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:18.420256 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:18.526270 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:18.526237 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:19.300550 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:19.300511 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:21.591701 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:32:21.591664 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:32:22.567340 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:22.567315 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh" Oct 30 01:32:22.589318 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:22.589297 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt" Oct 30 01:32:22.623623 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:22.623599 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19df0d00-62d5-4c6c-9cba-673cd02dbb8a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19df0d00-62d5-4c6c-9cba-673cd02dbb8a-plugins-config-volume\") pod \"19df0d00-62d5-4c6c-9cba-673cd02dbb8a\" (UID: \"19df0d00-62d5-4c6c-9cba-673cd02dbb8a\") " Oct 30 01:32:22.624087 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:22.623690 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa7f5966-fe14-4c2c-af88-ac82100209ba-kube-api-access-dgntw\" (UniqueName: \"kubernetes.io/projected/aa7f5966-fe14-4c2c-af88-ac82100209ba-kube-api-access-dgntw\") pod \"aa7f5966-fe14-4c2c-af88-ac82100209ba\" (UID: \"aa7f5966-fe14-4c2c-af88-ac82100209ba\") " Oct 30 01:32:22.624087 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:22.623810 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19df0d00-62d5-4c6c-9cba-673cd02dbb8a-kube-api-access-b27lj\" (UniqueName: \"kubernetes.io/projected/19df0d00-62d5-4c6c-9cba-673cd02dbb8a-kube-api-access-b27lj\") pod \"19df0d00-62d5-4c6c-9cba-673cd02dbb8a\" (UID: \"19df0d00-62d5-4c6c-9cba-673cd02dbb8a\") " Oct 30 01:32:22.624087 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:22.623919 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa7f5966-fe14-4c2c-af88-ac82100209ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa7f5966-fe14-4c2c-af88-ac82100209ba-plugins-config-volume\") pod \"aa7f5966-fe14-4c2c-af88-ac82100209ba\" (UID: \"aa7f5966-fe14-4c2c-af88-ac82100209ba\") " Oct 30 01:32:22.624087 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:22.624047 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19df0d00-62d5-4c6c-9cba-673cd02dbb8a-plugins-config-volume" pod "19df0d00-62d5-4c6c-9cba-673cd02dbb8a" (UID: "19df0d00-62d5-4c6c-9cba-673cd02dbb8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:22.624309 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:22.624289 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa7f5966-fe14-4c2c-af88-ac82100209ba-plugins-config-volume" pod "aa7f5966-fe14-4c2c-af88-ac82100209ba" (UID: "aa7f5966-fe14-4c2c-af88-ac82100209ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:22.624452 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:22.624429 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa7f5966-fe14-4c2c-af88-ac82100209ba-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:22.624515 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:22.624456 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19df0d00-62d5-4c6c-9cba-673cd02dbb8a-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:22.626028 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:22.626006 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7f5966-fe14-4c2c-af88-ac82100209ba-kube-api-access-dgntw" pod "aa7f5966-fe14-4c2c-af88-ac82100209ba" (UID: "aa7f5966-fe14-4c2c-af88-ac82100209ba"). InnerVolumeSpecName "kube-api-access-dgntw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:22.626166 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:22.626037 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19df0d00-62d5-4c6c-9cba-673cd02dbb8a-kube-api-access-b27lj" pod "19df0d00-62d5-4c6c-9cba-673cd02dbb8a" (UID: "19df0d00-62d5-4c6c-9cba-673cd02dbb8a"). InnerVolumeSpecName "kube-api-access-b27lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:22.725258 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:22.725224 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b27lj\" (UniqueName: \"kubernetes.io/projected/19df0d00-62d5-4c6c-9cba-673cd02dbb8a-kube-api-access-b27lj\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:22.725258 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:22.725253 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgntw\" (UniqueName: \"kubernetes.io/projected/aa7f5966-fe14-4c2c-af88-ac82100209ba-kube-api-access-dgntw\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:23.109509 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:23.109370 5300 generic.go:356] "Generic (PLEG): container finished" podID="aa7f5966-fe14-4c2c-af88-ac82100209ba" containerID="0cd561fc955ec100b8d6ddcd2a858e9c4c5004e428c9235caa8e915b013486c7" exitCode=0 Oct 30 01:32:23.109509 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:23.109428 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh" Oct 30 01:32:23.109509 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:23.109458 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh" event={"ID":"aa7f5966-fe14-4c2c-af88-ac82100209ba","Type":"ContainerDied","Data":"0cd561fc955ec100b8d6ddcd2a858e9c4c5004e428c9235caa8e915b013486c7"} Oct 30 01:32:23.136341 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:23.109525 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh" event={"ID":"aa7f5966-fe14-4c2c-af88-ac82100209ba","Type":"ContainerDied","Data":"bb33d4402f770124924258fc4ea02492930d59488e4dd4de75e49ba2429c0ebc"} Oct 30 01:32:23.136341 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:23.109564 5300 scope.go:122] "RemoveContainer" containerID="0cd561fc955ec100b8d6ddcd2a858e9c4c5004e428c9235caa8e915b013486c7" Oct 30 01:32:23.136341 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:23.110930 5300 generic.go:356] "Generic (PLEG): container finished" podID="19df0d00-62d5-4c6c-9cba-673cd02dbb8a" containerID="3d1bc8568f9354600975d6e738079c2dfba0e59d31b12455e7ed2b14bd1e2b6c" exitCode=0 Oct 30 01:32:23.136341 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:23.110978 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt" Oct 30 01:32:23.136341 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:23.111017 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt" event={"ID":"19df0d00-62d5-4c6c-9cba-673cd02dbb8a","Type":"ContainerDied","Data":"3d1bc8568f9354600975d6e738079c2dfba0e59d31b12455e7ed2b14bd1e2b6c"} Oct 30 01:32:23.136341 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:23.111059 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt" event={"ID":"19df0d00-62d5-4c6c-9cba-673cd02dbb8a","Type":"ContainerDied","Data":"2467892c11193f03857f979ca0627612f097022297fa3f203f25e193ad069c3e"} Oct 30 01:32:23.136341 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:23.119166 5300 scope.go:122] "RemoveContainer" containerID="0cd561fc955ec100b8d6ddcd2a858e9c4c5004e428c9235caa8e915b013486c7" Oct 30 01:32:23.136341 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:32:23.119446 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cd561fc955ec100b8d6ddcd2a858e9c4c5004e428c9235caa8e915b013486c7\": container with ID starting with 0cd561fc955ec100b8d6ddcd2a858e9c4c5004e428c9235caa8e915b013486c7 not found: ID does not exist" containerID="0cd561fc955ec100b8d6ddcd2a858e9c4c5004e428c9235caa8e915b013486c7" Oct 30 01:32:23.136341 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:23.119472 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cd561fc955ec100b8d6ddcd2a858e9c4c5004e428c9235caa8e915b013486c7"} err="failed to get container status \"0cd561fc955ec100b8d6ddcd2a858e9c4c5004e428c9235caa8e915b013486c7\": rpc error: code = NotFound desc = could not find container \"0cd561fc955ec100b8d6ddcd2a858e9c4c5004e428c9235caa8e915b013486c7\": container with ID starting with 0cd561fc955ec100b8d6ddcd2a858e9c4c5004e428c9235caa8e915b013486c7 not found: ID does not exist" Oct 30 01:32:23.136341 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:23.119498 5300 scope.go:122] "RemoveContainer" containerID="3d1bc8568f9354600975d6e738079c2dfba0e59d31b12455e7ed2b14bd1e2b6c" Oct 30 01:32:23.136341 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:23.127699 5300 scope.go:122] "RemoveContainer" containerID="3d1bc8568f9354600975d6e738079c2dfba0e59d31b12455e7ed2b14bd1e2b6c" Oct 30 01:32:23.136341 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:32:23.127947 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d1bc8568f9354600975d6e738079c2dfba0e59d31b12455e7ed2b14bd1e2b6c\": container with ID starting with 3d1bc8568f9354600975d6e738079c2dfba0e59d31b12455e7ed2b14bd1e2b6c not found: ID does not exist" containerID="3d1bc8568f9354600975d6e738079c2dfba0e59d31b12455e7ed2b14bd1e2b6c" Oct 30 01:32:23.136341 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:23.127970 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d1bc8568f9354600975d6e738079c2dfba0e59d31b12455e7ed2b14bd1e2b6c"} err="failed to get container status \"3d1bc8568f9354600975d6e738079c2dfba0e59d31b12455e7ed2b14bd1e2b6c\": rpc error: code = NotFound desc = could not find container \"3d1bc8568f9354600975d6e738079c2dfba0e59d31b12455e7ed2b14bd1e2b6c\": container with ID starting with 3d1bc8568f9354600975d6e738079c2dfba0e59d31b12455e7ed2b14bd1e2b6c not found: ID does not exist" Oct 30 01:32:23.165580 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:23.165553 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt"] Oct 30 01:32:23.179433 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:23.179413 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-e92648-7965cdbb99-kfbbt"] Oct 30 01:32:23.203220 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:23.203188 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh"] Oct 30 01:32:23.204979 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:23.204961 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-e92648-57477cbddb-fhpwh"] Oct 30 01:32:23.282852 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:23.282814 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9" podUID="8fd0d5d9-adcd-4d24-8916-379be6b844e9" containerName="payload-processing" containerID="cri-o://e9a5a8c0cb1cc9df7107e30dbb93dd8890e435a01e36f0c4ee7f5c82b5c69c75" gracePeriod=25 Oct 30 01:32:23.800908 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:23.800872 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:23.820515 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:23.820482 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19df0d00-62d5-4c6c-9cba-673cd02dbb8a" path="/var/lib/kubelet/pods/19df0d00-62d5-4c6c-9cba-673cd02dbb8a/volumes" Oct 30 01:32:23.820975 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:23.820954 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa7f5966-fe14-4c2c-af88-ac82100209ba" path="/var/lib/kubelet/pods/aa7f5966-fe14-4c2c-af88-ac82100209ba/volumes" Oct 30 01:32:23.901895 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:23.901846 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:24.158700 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:24.158612 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:25.889474 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:25.889440 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9" podUID="8fd0d5d9-adcd-4d24-8916-379be6b844e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:32:26.092027 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:26.091995 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:26.298890 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:26.298858 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:26.990965 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:26.990900 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:27.679789 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:27.679735 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerName="main" containerID="cri-o://baf22d4273bdbf376eda851e14ea5ce738e7021ef495d6b332aa8121f113d245" gracePeriod=45 Oct 30 01:32:27.712874 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:27.712842 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" containerID="cri-o://f2f96c802581b72e7deb54df346e854e292c345b6d55d991851c1552a4596760" gracePeriod=45 Oct 30 01:32:27.961127 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:27.961103 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 01:32:27.985844 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:27.985821 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 01:32:28.082671 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.082643 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-dshm\") pod \"a36b0810-b564-4c4f-900f-6ec73fabd405\" (UID: \"a36b0810-b564-4c4f-900f-6ec73fabd405\") " Oct 30 01:32:28.083197 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.082682 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-home\" (UniqueName: \"kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-home\") pod \"a36b0810-b564-4c4f-900f-6ec73fabd405\" (UID: \"a36b0810-b564-4c4f-900f-6ec73fabd405\") " Oct 30 01:32:28.083197 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.082713 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a36b0810-b564-4c4f-900f-6ec73fabd405-tls-certs\" (UniqueName: \"kubernetes.io/secret/a36b0810-b564-4c4f-900f-6ec73fabd405-tls-certs\") pod \"a36b0810-b564-4c4f-900f-6ec73fabd405\" (UID: \"a36b0810-b564-4c4f-900f-6ec73fabd405\") " Oct 30 01:32:28.083197 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.082753 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a36b0810-b564-4c4f-900f-6ec73fabd405-kube-api-access-t8sfq\" (UniqueName: \"kubernetes.io/projected/a36b0810-b564-4c4f-900f-6ec73fabd405-kube-api-access-t8sfq\") pod \"a36b0810-b564-4c4f-900f-6ec73fabd405\" (UID: \"a36b0810-b564-4c4f-900f-6ec73fabd405\") " Oct 30 01:32:28.083197 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.082822 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-model-cache\") pod \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\" (UID: \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\") " Oct 30 01:32:28.083197 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.082856 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-tls-certs\" (UniqueName: \"kubernetes.io/secret/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-tls-certs\") pod \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\" (UID: \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\") " Oct 30 01:32:28.083197 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.082882 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-model-cache\") pod \"a36b0810-b564-4c4f-900f-6ec73fabd405\" (UID: \"a36b0810-b564-4c4f-900f-6ec73fabd405\") " Oct 30 01:32:28.083197 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.082961 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-kube-api-access-z8h9n\" (UniqueName: \"kubernetes.io/projected/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-kube-api-access-z8h9n\") pod \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\" (UID: \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\") " Oct 30 01:32:28.083197 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.083001 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-dshm\") pod \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\" (UID: \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\") " Oct 30 01:32:28.083197 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.083066 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-home" pod "a36b0810-b564-4c4f-900f-6ec73fabd405" (UID: "a36b0810-b564-4c4f-900f-6ec73fabd405"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:28.083197 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.083082 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-tmp-dir\") pod \"a36b0810-b564-4c4f-900f-6ec73fabd405\" (UID: \"a36b0810-b564-4c4f-900f-6ec73fabd405\") " Oct 30 01:32:28.083197 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.083148 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-home\" (UniqueName: \"kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-home\") pod \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\" (UID: \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\") " Oct 30 01:32:28.083197 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.083188 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-tmp-dir\") pod \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\" (UID: \"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7\") " Oct 30 01:32:28.083928 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.083281 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-model-cache" pod "a36b0810-b564-4c4f-900f-6ec73fabd405" (UID: "a36b0810-b564-4c4f-900f-6ec73fabd405"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:28.083928 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.083480 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-model-cache" pod "e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" (UID: "e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:28.083928 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.083673 5300 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-home\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:28.083928 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.083686 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-tmp-dir" pod "e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" (UID: "e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:28.083928 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.083692 5300 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-model-cache\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:28.083928 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.083731 5300 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-model-cache\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:28.084264 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.083956 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-tmp-dir" pod "a36b0810-b564-4c4f-900f-6ec73fabd405" (UID: "a36b0810-b564-4c4f-900f-6ec73fabd405"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:28.084264 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.084142 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-home" pod "e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" (UID: "e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:28.086289 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.086256 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-dshm" pod "a36b0810-b564-4c4f-900f-6ec73fabd405" (UID: "a36b0810-b564-4c4f-900f-6ec73fabd405"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:28.086445 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.086405 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-tls-certs" pod "e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" (UID: "e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:28.086691 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.086671 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a36b0810-b564-4c4f-900f-6ec73fabd405-tls-certs" pod "a36b0810-b564-4c4f-900f-6ec73fabd405" (UID: "a36b0810-b564-4c4f-900f-6ec73fabd405"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:28.086816 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.086709 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-kube-api-access-z8h9n" pod "e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" (UID: "e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7"). InnerVolumeSpecName "kube-api-access-z8h9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:28.087065 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.087046 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-dshm" pod "e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" (UID: "e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:28.087143 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.087105 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a36b0810-b564-4c4f-900f-6ec73fabd405-kube-api-access-t8sfq" pod "a36b0810-b564-4c4f-900f-6ec73fabd405" (UID: "a36b0810-b564-4c4f-900f-6ec73fabd405"). InnerVolumeSpecName "kube-api-access-t8sfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:28.143694 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.143664 5300 generic.go:356] "Generic (PLEG): container finished" podID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerID="f2f96c802581b72e7deb54df346e854e292c345b6d55d991851c1552a4596760" exitCode=0 Oct 30 01:32:28.143872 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.143698 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" event={"ID":"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7","Type":"ContainerDied","Data":"f2f96c802581b72e7deb54df346e854e292c345b6d55d991851c1552a4596760"} Oct 30 01:32:28.143872 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.143718 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" Oct 30 01:32:28.143872 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.143742 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct" event={"ID":"e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7","Type":"ContainerDied","Data":"226026643f758e2df9dbc3de29ff2da11584339cf435cb4e320ec7be5798f8f2"} Oct 30 01:32:28.143872 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.143790 5300 scope.go:122] "RemoveContainer" containerID="f2f96c802581b72e7deb54df346e854e292c345b6d55d991851c1552a4596760" Oct 30 01:32:28.145244 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.145220 5300 generic.go:356] "Generic (PLEG): container finished" podID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerID="baf22d4273bdbf376eda851e14ea5ce738e7021ef495d6b332aa8121f113d245" exitCode=0 Oct 30 01:32:28.145367 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.145257 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" Oct 30 01:32:28.145367 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.145274 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" event={"ID":"a36b0810-b564-4c4f-900f-6ec73fabd405","Type":"ContainerDied","Data":"baf22d4273bdbf376eda851e14ea5ce738e7021ef495d6b332aa8121f113d245"} Oct 30 01:32:28.145367 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.145319 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4" event={"ID":"a36b0810-b564-4c4f-900f-6ec73fabd405","Type":"ContainerDied","Data":"0036fc8b6874ab253986a1844e418b919b35e71a75f15889a5993b261b545a5c"} Oct 30 01:32:28.153677 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.153656 5300 scope.go:122] "RemoveContainer" containerID="f2f96c802581b72e7deb54df346e854e292c345b6d55d991851c1552a4596760" Oct 30 01:32:28.154013 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:32:28.153931 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2f96c802581b72e7deb54df346e854e292c345b6d55d991851c1552a4596760\": container with ID starting with f2f96c802581b72e7deb54df346e854e292c345b6d55d991851c1552a4596760 not found: ID does not exist" containerID="f2f96c802581b72e7deb54df346e854e292c345b6d55d991851c1552a4596760" Oct 30 01:32:28.154013 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.153958 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2f96c802581b72e7deb54df346e854e292c345b6d55d991851c1552a4596760"} err="failed to get container status \"f2f96c802581b72e7deb54df346e854e292c345b6d55d991851c1552a4596760\": rpc error: code = NotFound desc = could not find container \"f2f96c802581b72e7deb54df346e854e292c345b6d55d991851c1552a4596760\": container with ID starting with f2f96c802581b72e7deb54df346e854e292c345b6d55d991851c1552a4596760 not found: ID does not exist" Oct 30 01:32:28.154013 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.153974 5300 scope.go:122] "RemoveContainer" containerID="baf22d4273bdbf376eda851e14ea5ce738e7021ef495d6b332aa8121f113d245" Oct 30 01:32:28.161870 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.161855 5300 scope.go:122] "RemoveContainer" containerID="baf22d4273bdbf376eda851e14ea5ce738e7021ef495d6b332aa8121f113d245" Oct 30 01:32:28.162132 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:32:28.162111 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baf22d4273bdbf376eda851e14ea5ce738e7021ef495d6b332aa8121f113d245\": container with ID starting with baf22d4273bdbf376eda851e14ea5ce738e7021ef495d6b332aa8121f113d245 not found: ID does not exist" containerID="baf22d4273bdbf376eda851e14ea5ce738e7021ef495d6b332aa8121f113d245" Oct 30 01:32:28.162203 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.162137 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baf22d4273bdbf376eda851e14ea5ce738e7021ef495d6b332aa8121f113d245"} err="failed to get container status \"baf22d4273bdbf376eda851e14ea5ce738e7021ef495d6b332aa8121f113d245\": rpc error: code = NotFound desc = could not find container \"baf22d4273bdbf376eda851e14ea5ce738e7021ef495d6b332aa8121f113d245\": container with ID starting with baf22d4273bdbf376eda851e14ea5ce738e7021ef495d6b332aa8121f113d245 not found: ID does not exist" Oct 30 01:32:28.184855 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.184835 5300 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-tls-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:28.184855 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.184858 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8h9n\" (UniqueName: \"kubernetes.io/projected/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-kube-api-access-z8h9n\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:28.184990 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.184868 5300 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-dshm\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:28.184990 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.184878 5300 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-tmp-dir\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:28.184990 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.184887 5300 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-home\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:28.184990 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.184894 5300 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7-tmp-dir\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:28.184990 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.184901 5300 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a36b0810-b564-4c4f-900f-6ec73fabd405-dshm\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:28.184990 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.184908 5300 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a36b0810-b564-4c4f-900f-6ec73fabd405-tls-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:28.184990 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.184916 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8sfq\" (UniqueName: \"kubernetes.io/projected/a36b0810-b564-4c4f-900f-6ec73fabd405-kube-api-access-t8sfq\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:28.199100 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.199047 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4"] Oct 30 01:32:28.203144 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.203119 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-c4847c/distinct-2-w6-c4847c-kserve-7554595bfc-b8xs4"] Oct 30 01:32:28.239807 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.239784 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct"] Oct 30 01:32:28.243687 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:28.243668 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-c4847c/simulator-w6-c4847c-kserve-56bb64f4fd-6wrct"] Oct 30 01:32:29.818976 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:29.818944 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" path="/var/lib/kubelet/pods/a36b0810-b564-4c4f-900f-6ec73fabd405/volumes" Oct 30 01:32:29.819507 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:29.819486 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" path="/var/lib/kubelet/pods/e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7/volumes" Oct 30 01:32:32.432989 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:32.432955 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:35.698124 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:35.698091 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:35.889452 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:35.889409 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9" podUID="8fd0d5d9-adcd-4d24-8916-379be6b844e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:32:35.928721 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:35.928684 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:36.364246 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:36.364211 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:36.466356 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:36.466320 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:36.989860 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:36.989828 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:42.658548 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:42.658511 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:42.758277 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:42.758244 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:42.958603 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:42.958504 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq"] Oct 30 01:32:42.958831 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:42.958803 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" podUID="e7d2e040-342a-4ea4-a23d-c36722f2d64e" containerName="istio-proxy" containerID="cri-o://7179cad9c68a74a0dd3340ec7249dd966f39f34e243f2b71c3fe29e7aebc9446" gracePeriod=30 Oct 30 01:32:42.993418 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:42.993389 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:43.532124 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:43.532097 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9" Oct 30 01:32:43.557593 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:43.557539 5300 patch_prober.go:29] interesting pod/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:32:43.557701 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:43.557587 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" podUID="e7d2e040-342a-4ea4-a23d-c36722f2d64e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:32:43.643742 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:43.643717 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fd0d5d9-adcd-4d24-8916-379be6b844e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fd0d5d9-adcd-4d24-8916-379be6b844e9-plugins-config-volume\") pod \"8fd0d5d9-adcd-4d24-8916-379be6b844e9\" (UID: \"8fd0d5d9-adcd-4d24-8916-379be6b844e9\") " Oct 30 01:32:43.643893 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:43.643824 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fd0d5d9-adcd-4d24-8916-379be6b844e9-kube-api-access-w5rjn\" (UniqueName: \"kubernetes.io/projected/8fd0d5d9-adcd-4d24-8916-379be6b844e9-kube-api-access-w5rjn\") pod \"8fd0d5d9-adcd-4d24-8916-379be6b844e9\" (UID: \"8fd0d5d9-adcd-4d24-8916-379be6b844e9\") " Oct 30 01:32:43.644146 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:43.644125 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fd0d5d9-adcd-4d24-8916-379be6b844e9-plugins-config-volume" pod "8fd0d5d9-adcd-4d24-8916-379be6b844e9" (UID: "8fd0d5d9-adcd-4d24-8916-379be6b844e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:43.645938 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:43.645917 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fd0d5d9-adcd-4d24-8916-379be6b844e9-kube-api-access-w5rjn" pod "8fd0d5d9-adcd-4d24-8916-379be6b844e9" (UID: "8fd0d5d9-adcd-4d24-8916-379be6b844e9"). InnerVolumeSpecName "kube-api-access-w5rjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:43.744917 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:43.744890 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fd0d5d9-adcd-4d24-8916-379be6b844e9-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:43.744917 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:43.744913 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5rjn\" (UniqueName: \"kubernetes.io/projected/8fd0d5d9-adcd-4d24-8916-379be6b844e9-kube-api-access-w5rjn\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:44.224709 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:44.224668 5300 generic.go:356] "Generic (PLEG): container finished" podID="8fd0d5d9-adcd-4d24-8916-379be6b844e9" containerID="e9a5a8c0cb1cc9df7107e30dbb93dd8890e435a01e36f0c4ee7f5c82b5c69c75" exitCode=0 Oct 30 01:32:44.224916 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:44.224725 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9" Oct 30 01:32:44.224916 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:44.224779 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9" event={"ID":"8fd0d5d9-adcd-4d24-8916-379be6b844e9","Type":"ContainerDied","Data":"e9a5a8c0cb1cc9df7107e30dbb93dd8890e435a01e36f0c4ee7f5c82b5c69c75"} Oct 30 01:32:44.224916 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:44.224843 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9" event={"ID":"8fd0d5d9-adcd-4d24-8916-379be6b844e9","Type":"ContainerDied","Data":"3f606b19a4b2ef983e7fa00d1c3488b0372ff3d23aac1dc195556f26dd314b81"} Oct 30 01:32:44.224916 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:44.224867 5300 scope.go:122] "RemoveContainer" containerID="e9a5a8c0cb1cc9df7107e30dbb93dd8890e435a01e36f0c4ee7f5c82b5c69c75" Oct 30 01:32:44.233797 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:44.233775 5300 scope.go:122] "RemoveContainer" containerID="e9a5a8c0cb1cc9df7107e30dbb93dd8890e435a01e36f0c4ee7f5c82b5c69c75" Oct 30 01:32:44.234075 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:32:44.234053 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9a5a8c0cb1cc9df7107e30dbb93dd8890e435a01e36f0c4ee7f5c82b5c69c75\": container with ID starting with e9a5a8c0cb1cc9df7107e30dbb93dd8890e435a01e36f0c4ee7f5c82b5c69c75 not found: ID does not exist" containerID="e9a5a8c0cb1cc9df7107e30dbb93dd8890e435a01e36f0c4ee7f5c82b5c69c75" Oct 30 01:32:44.234142 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:44.234083 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a5a8c0cb1cc9df7107e30dbb93dd8890e435a01e36f0c4ee7f5c82b5c69c75"} err="failed to get container status \"e9a5a8c0cb1cc9df7107e30dbb93dd8890e435a01e36f0c4ee7f5c82b5c69c75\": rpc error: code = NotFound desc = could not find container \"e9a5a8c0cb1cc9df7107e30dbb93dd8890e435a01e36f0c4ee7f5c82b5c69c75\": container with ID starting with e9a5a8c0cb1cc9df7107e30dbb93dd8890e435a01e36f0c4ee7f5c82b5c69c75 not found: ID does not exist" Oct 30 01:32:44.282078 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:44.282053 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9"] Oct 30 01:32:44.295863 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:44.295838 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-c4847c-65f8bb6c56-hd5f9"] Oct 30 01:32:45.818801 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:45.818745 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fd0d5d9-adcd-4d24-8916-379be6b844e9" path="/var/lib/kubelet/pods/8fd0d5d9-adcd-4d24-8916-379be6b844e9/volumes" Oct 30 01:32:47.676737 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:47.676703 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59cfcff4f-mxd2h"] Oct 30 01:32:47.677226 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:47.676922 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-59cfcff4f-mxd2h" podUID="76bdf51b-ab36-40c2-9284-6cd0e71f1457" containerName="manager" containerID="cri-o://1ccea0c94c41cb78b0a0c99a6cf719917fe14a5ac9094ba0b7fd5bdbb71380d7" gracePeriod=30 Oct 30 01:32:47.945433 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:47.945399 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59cfcff4f-mxd2h" Oct 30 01:32:48.084071 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.084041 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76bdf51b-ab36-40c2-9284-6cd0e71f1457-webhook-cert\" (UniqueName: \"kubernetes.io/secret/76bdf51b-ab36-40c2-9284-6cd0e71f1457-webhook-cert\") pod \"76bdf51b-ab36-40c2-9284-6cd0e71f1457\" (UID: \"76bdf51b-ab36-40c2-9284-6cd0e71f1457\") " Oct 30 01:32:48.084236 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.084095 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76bdf51b-ab36-40c2-9284-6cd0e71f1457-kube-api-access-m5f6c\" (UniqueName: \"kubernetes.io/projected/76bdf51b-ab36-40c2-9284-6cd0e71f1457-kube-api-access-m5f6c\") pod \"76bdf51b-ab36-40c2-9284-6cd0e71f1457\" (UID: \"76bdf51b-ab36-40c2-9284-6cd0e71f1457\") " Oct 30 01:32:48.084236 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.084135 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76bdf51b-ab36-40c2-9284-6cd0e71f1457-metrics-certs\" (UniqueName: \"kubernetes.io/secret/76bdf51b-ab36-40c2-9284-6cd0e71f1457-metrics-certs\") pod \"76bdf51b-ab36-40c2-9284-6cd0e71f1457\" (UID: \"76bdf51b-ab36-40c2-9284-6cd0e71f1457\") " Oct 30 01:32:48.086754 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.086721 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76bdf51b-ab36-40c2-9284-6cd0e71f1457-webhook-cert" pod "76bdf51b-ab36-40c2-9284-6cd0e71f1457" (UID: "76bdf51b-ab36-40c2-9284-6cd0e71f1457"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:48.086984 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.086962 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76bdf51b-ab36-40c2-9284-6cd0e71f1457-metrics-certs" pod "76bdf51b-ab36-40c2-9284-6cd0e71f1457" (UID: "76bdf51b-ab36-40c2-9284-6cd0e71f1457"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:48.086984 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.086967 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76bdf51b-ab36-40c2-9284-6cd0e71f1457-kube-api-access-m5f6c" pod "76bdf51b-ab36-40c2-9284-6cd0e71f1457" (UID: "76bdf51b-ab36-40c2-9284-6cd0e71f1457"). InnerVolumeSpecName "kube-api-access-m5f6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:48.186296 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.186208 5300 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76bdf51b-ab36-40c2-9284-6cd0e71f1457-webhook-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:48.186296 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.186241 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5f6c\" (UniqueName: \"kubernetes.io/projected/76bdf51b-ab36-40c2-9284-6cd0e71f1457-kube-api-access-m5f6c\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:48.186296 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.186254 5300 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76bdf51b-ab36-40c2-9284-6cd0e71f1457-metrics-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:48.214454 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.214429 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 01:32:48.250824 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.250790 5300 generic.go:356] "Generic (PLEG): container finished" podID="76bdf51b-ab36-40c2-9284-6cd0e71f1457" containerID="1ccea0c94c41cb78b0a0c99a6cf719917fe14a5ac9094ba0b7fd5bdbb71380d7" exitCode=0 Oct 30 01:32:48.250824 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.250825 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59cfcff4f-mxd2h" Oct 30 01:32:48.251105 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.250876 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59cfcff4f-mxd2h" event={"ID":"76bdf51b-ab36-40c2-9284-6cd0e71f1457","Type":"ContainerDied","Data":"1ccea0c94c41cb78b0a0c99a6cf719917fe14a5ac9094ba0b7fd5bdbb71380d7"} Oct 30 01:32:48.251105 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.250935 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59cfcff4f-mxd2h" event={"ID":"76bdf51b-ab36-40c2-9284-6cd0e71f1457","Type":"ContainerDied","Data":"1af067d3720eaa15aeae1116fefa7b542f4d51e121383bc72e90eb83b9ccbaf6"} Oct 30 01:32:48.251105 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.250964 5300 scope.go:122] "RemoveContainer" containerID="1ccea0c94c41cb78b0a0c99a6cf719917fe14a5ac9094ba0b7fd5bdbb71380d7" Oct 30 01:32:48.256850 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.253718 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" Oct 30 01:32:48.256850 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.253839 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" event={"ID":"e7d2e040-342a-4ea4-a23d-c36722f2d64e","Type":"ContainerDied","Data":"7179cad9c68a74a0dd3340ec7249dd966f39f34e243f2b71c3fe29e7aebc9446"} Oct 30 01:32:48.256850 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.253992 5300 generic.go:356] "Generic (PLEG): container finished" podID="e7d2e040-342a-4ea4-a23d-c36722f2d64e" containerID="7179cad9c68a74a0dd3340ec7249dd966f39f34e243f2b71c3fe29e7aebc9446" exitCode=0 Oct 30 01:32:48.256850 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.254021 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq" event={"ID":"e7d2e040-342a-4ea4-a23d-c36722f2d64e","Type":"ContainerDied","Data":"a0fa3bf0588a27cdb98be01cc394038d8009ed74d55077f522f6236e88cdec66"} Oct 30 01:32:48.279460 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.279427 5300 scope.go:122] "RemoveContainer" containerID="1ccea0c94c41cb78b0a0c99a6cf719917fe14a5ac9094ba0b7fd5bdbb71380d7" Oct 30 01:32:48.290321 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:32:48.290236 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ccea0c94c41cb78b0a0c99a6cf719917fe14a5ac9094ba0b7fd5bdbb71380d7\": container with ID starting with 1ccea0c94c41cb78b0a0c99a6cf719917fe14a5ac9094ba0b7fd5bdbb71380d7 not found: ID does not exist" containerID="1ccea0c94c41cb78b0a0c99a6cf719917fe14a5ac9094ba0b7fd5bdbb71380d7" Oct 30 01:32:48.290321 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.290299 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ccea0c94c41cb78b0a0c99a6cf719917fe14a5ac9094ba0b7fd5bdbb71380d7"} err="failed to get container status \"1ccea0c94c41cb78b0a0c99a6cf719917fe14a5ac9094ba0b7fd5bdbb71380d7\": rpc error: code = NotFound desc = could not find container \"1ccea0c94c41cb78b0a0c99a6cf719917fe14a5ac9094ba0b7fd5bdbb71380d7\": container with ID starting with 1ccea0c94c41cb78b0a0c99a6cf719917fe14a5ac9094ba0b7fd5bdbb71380d7 not found: ID does not exist" Oct 30 01:32:48.290321 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.290331 5300 scope.go:122] "RemoveContainer" containerID="7179cad9c68a74a0dd3340ec7249dd966f39f34e243f2b71c3fe29e7aebc9446" Oct 30 01:32:48.302517 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.302491 5300 scope.go:122] "RemoveContainer" containerID="7179cad9c68a74a0dd3340ec7249dd966f39f34e243f2b71c3fe29e7aebc9446" Oct 30 01:32:48.302877 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:32:48.302841 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7179cad9c68a74a0dd3340ec7249dd966f39f34e243f2b71c3fe29e7aebc9446\": container with ID starting with 7179cad9c68a74a0dd3340ec7249dd966f39f34e243f2b71c3fe29e7aebc9446 not found: ID does not exist" containerID="7179cad9c68a74a0dd3340ec7249dd966f39f34e243f2b71c3fe29e7aebc9446" Oct 30 01:32:48.302973 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.302879 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7179cad9c68a74a0dd3340ec7249dd966f39f34e243f2b71c3fe29e7aebc9446"} err="failed to get container status \"7179cad9c68a74a0dd3340ec7249dd966f39f34e243f2b71c3fe29e7aebc9446\": rpc error: code = NotFound desc = could not find container \"7179cad9c68a74a0dd3340ec7249dd966f39f34e243f2b71c3fe29e7aebc9446\": container with ID starting with 7179cad9c68a74a0dd3340ec7249dd966f39f34e243f2b71c3fe29e7aebc9446 not found: ID does not exist" Oct 30 01:32:48.336650 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.336604 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59cfcff4f-mxd2h"] Oct 30 01:32:48.343067 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.343033 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-59cfcff4f-mxd2h"] Oct 30 01:32:48.391310 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.391273 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-workload-socket\") pod \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " Oct 30 01:32:48.391495 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.391331 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-podinfo\") pod \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " Oct 30 01:32:48.391495 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.391370 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-envoy\") pod \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " Oct 30 01:32:48.391495 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.391417 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7d2e040-342a-4ea4-a23d-c36722f2d64e-kube-api-access-h6m2p\" (UniqueName: \"kubernetes.io/projected/e7d2e040-342a-4ea4-a23d-c36722f2d64e-kube-api-access-h6m2p\") pod \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " Oct 30 01:32:48.391495 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.391457 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-token\" (UniqueName: \"kubernetes.io/projected/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-token\") pod \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " Oct 30 01:32:48.391731 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.391503 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-credential-socket\") pod \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " Oct 30 01:32:48.391731 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.391534 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-workload-certs\") pod \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " Oct 30 01:32:48.391731 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.391589 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istiod-ca-cert\") pod \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " Oct 30 01:32:48.391731 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.391629 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-data\") pod \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\" (UID: \"e7d2e040-342a-4ea4-a23d-c36722f2d64e\") " Oct 30 01:32:48.391973 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.391918 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-workload-certs" pod "e7d2e040-342a-4ea4-a23d-c36722f2d64e" (UID: "e7d2e040-342a-4ea4-a23d-c36722f2d64e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:48.392036 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.392016 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:48.392160 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.392114 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-credential-socket" pod "e7d2e040-342a-4ea4-a23d-c36722f2d64e" (UID: "e7d2e040-342a-4ea4-a23d-c36722f2d64e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:48.392284 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.392258 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-workload-socket" pod "e7d2e040-342a-4ea4-a23d-c36722f2d64e" (UID: "e7d2e040-342a-4ea4-a23d-c36722f2d64e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:48.392397 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.392280 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-data" pod "e7d2e040-342a-4ea4-a23d-c36722f2d64e" (UID: "e7d2e040-342a-4ea4-a23d-c36722f2d64e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:48.392550 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.392339 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istiod-ca-cert" pod "e7d2e040-342a-4ea4-a23d-c36722f2d64e" (UID: "e7d2e040-342a-4ea4-a23d-c36722f2d64e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:48.393818 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.393792 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d2e040-342a-4ea4-a23d-c36722f2d64e-kube-api-access-h6m2p" pod "e7d2e040-342a-4ea4-a23d-c36722f2d64e" (UID: "e7d2e040-342a-4ea4-a23d-c36722f2d64e"). InnerVolumeSpecName "kube-api-access-h6m2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:48.394047 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.394028 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-envoy" pod "e7d2e040-342a-4ea4-a23d-c36722f2d64e" (UID: "e7d2e040-342a-4ea4-a23d-c36722f2d64e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:32:48.394321 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.394299 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-token" pod "e7d2e040-342a-4ea4-a23d-c36722f2d64e" (UID: "e7d2e040-342a-4ea4-a23d-c36722f2d64e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:48.394444 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.394421 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-podinfo" pod "e7d2e040-342a-4ea4-a23d-c36722f2d64e" (UID: "e7d2e040-342a-4ea4-a23d-c36722f2d64e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:32:48.492920 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.492892 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:48.492920 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.492921 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:48.493161 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.492930 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:48.493161 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.492938 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:48.493161 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.492947 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:48.493161 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.492956 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6m2p\" (UniqueName: \"kubernetes.io/projected/e7d2e040-342a-4ea4-a23d-c36722f2d64e-kube-api-access-h6m2p\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:48.493161 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.492965 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7d2e040-342a-4ea4-a23d-c36722f2d64e-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:48.493161 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.492974 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7d2e040-342a-4ea4-a23d-c36722f2d64e-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:32:48.601472 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.601437 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq"] Oct 30 01:32:48.615602 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:48.615573 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c4847c-openshift-default-65bf677b7f-84snq"] Oct 30 01:32:49.779504 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:49.779472 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:49.821513 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:49.821478 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76bdf51b-ab36-40c2-9284-6cd0e71f1457" path="/var/lib/kubelet/pods/76bdf51b-ab36-40c2-9284-6cd0e71f1457/volumes" Oct 30 01:32:49.821973 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:49.821954 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7d2e040-342a-4ea4-a23d-c36722f2d64e" path="/var/lib/kubelet/pods/e7d2e040-342a-4ea4-a23d-c36722f2d64e/volumes" Oct 30 01:32:50.026378 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:50.026341 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:50.188936 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:50.188837 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:50.288538 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:50.288502 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:50.505689 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:50.505662 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:51.592413 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:32:51.592378 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:32:56.231677 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:32:56.231644 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:09.783822 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:33:09.783722 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:21.592260 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:33:21.592220 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:33:22.098088 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:33:22.098056 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:24.438904 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:33:24.438851 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:33.234734 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:33:33.234695 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:33.335959 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:33:33.335927 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:35.896690 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:33:35.896661 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:51.591129 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:33:51.591092 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:34:17.812274 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:17.812234 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj"] Oct 30 01:34:17.812899 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:17.812425 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" podUID="c8404a58-57e8-4ade-addc-c7b180433b35" containerName="discovery" containerID="cri-o://39fe6b535c6fbb35041bd132ddf410a95abb5ec5760ee9345bc23ffd4607d546" gracePeriod=30 Oct 30 01:34:17.909938 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:17.909891 5300 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-ppsvj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.167:8080/ready\": dial tcp 10.133.0.167:8080: connect: connection refused" start-of-body= Oct 30 01:34:17.910120 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:17.909962 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" podUID="c8404a58-57e8-4ade-addc-c7b180433b35" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.167:8080/ready\": dial tcp 10.133.0.167:8080: connect: connection refused" Oct 30 01:34:18.076311 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.076238 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" Oct 30 01:34:18.189345 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.189316 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8404a58-57e8-4ade-addc-c7b180433b35-kube-api-access-gxh4k\" (UniqueName: \"kubernetes.io/projected/c8404a58-57e8-4ade-addc-c7b180433b35-kube-api-access-gxh4k\") pod \"c8404a58-57e8-4ade-addc-c7b180433b35\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " Oct 30 01:34:18.189517 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.189406 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8404a58-57e8-4ade-addc-c7b180433b35-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8404a58-57e8-4ade-addc-c7b180433b35-istio-kubeconfig\") pod \"c8404a58-57e8-4ade-addc-c7b180433b35\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " Oct 30 01:34:18.189517 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.189451 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8404a58-57e8-4ade-addc-c7b180433b35-istio-token\" (UniqueName: \"kubernetes.io/projected/c8404a58-57e8-4ade-addc-c7b180433b35-istio-token\") pod \"c8404a58-57e8-4ade-addc-c7b180433b35\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " Oct 30 01:34:18.189517 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.189470 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8404a58-57e8-4ade-addc-c7b180433b35-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8404a58-57e8-4ade-addc-c7b180433b35-istio-csr-ca-configmap\") pod \"c8404a58-57e8-4ade-addc-c7b180433b35\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " Oct 30 01:34:18.189674 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.189539 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8404a58-57e8-4ade-addc-c7b180433b35-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8404a58-57e8-4ade-addc-c7b180433b35-local-certs\") pod \"c8404a58-57e8-4ade-addc-c7b180433b35\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " Oct 30 01:34:18.189674 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.189579 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8404a58-57e8-4ade-addc-c7b180433b35-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8404a58-57e8-4ade-addc-c7b180433b35-istio-csr-dns-cert\") pod \"c8404a58-57e8-4ade-addc-c7b180433b35\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " Oct 30 01:34:18.189674 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.189614 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8404a58-57e8-4ade-addc-c7b180433b35-cacerts\" (UniqueName: \"kubernetes.io/secret/c8404a58-57e8-4ade-addc-c7b180433b35-cacerts\") pod \"c8404a58-57e8-4ade-addc-c7b180433b35\" (UID: \"c8404a58-57e8-4ade-addc-c7b180433b35\") " Oct 30 01:34:18.190040 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.189980 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8404a58-57e8-4ade-addc-c7b180433b35-istio-csr-ca-configmap" pod "c8404a58-57e8-4ade-addc-c7b180433b35" (UID: "c8404a58-57e8-4ade-addc-c7b180433b35"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:34:18.191895 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.191792 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8404a58-57e8-4ade-addc-c7b180433b35-istio-token" pod "c8404a58-57e8-4ade-addc-c7b180433b35" (UID: "c8404a58-57e8-4ade-addc-c7b180433b35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:34:18.192032 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.191979 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8404a58-57e8-4ade-addc-c7b180433b35-local-certs" pod "c8404a58-57e8-4ade-addc-c7b180433b35" (UID: "c8404a58-57e8-4ade-addc-c7b180433b35"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:34:18.192032 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.191982 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8404a58-57e8-4ade-addc-c7b180433b35-kube-api-access-gxh4k" pod "c8404a58-57e8-4ade-addc-c7b180433b35" (UID: "c8404a58-57e8-4ade-addc-c7b180433b35"). InnerVolumeSpecName "kube-api-access-gxh4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:34:18.192300 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.192275 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8404a58-57e8-4ade-addc-c7b180433b35-istio-kubeconfig" pod "c8404a58-57e8-4ade-addc-c7b180433b35" (UID: "c8404a58-57e8-4ade-addc-c7b180433b35"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:34:18.192577 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.192555 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8404a58-57e8-4ade-addc-c7b180433b35-istio-csr-dns-cert" pod "c8404a58-57e8-4ade-addc-c7b180433b35" (UID: "c8404a58-57e8-4ade-addc-c7b180433b35"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:34:18.192668 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.192573 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8404a58-57e8-4ade-addc-c7b180433b35-cacerts" pod "c8404a58-57e8-4ade-addc-c7b180433b35" (UID: "c8404a58-57e8-4ade-addc-c7b180433b35"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:34:18.290914 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.290881 5300 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c8404a58-57e8-4ade-addc-c7b180433b35-cacerts\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:34:18.290914 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.290912 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxh4k\" (UniqueName: \"kubernetes.io/projected/c8404a58-57e8-4ade-addc-c7b180433b35-kube-api-access-gxh4k\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:34:18.291120 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.290922 5300 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c8404a58-57e8-4ade-addc-c7b180433b35-istio-kubeconfig\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:34:18.291120 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.290932 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8404a58-57e8-4ade-addc-c7b180433b35-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:34:18.291120 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.290940 5300 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c8404a58-57e8-4ade-addc-c7b180433b35-istio-csr-ca-configmap\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:34:18.291120 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.290948 5300 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8404a58-57e8-4ade-addc-c7b180433b35-local-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:34:18.291120 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.290956 5300 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c8404a58-57e8-4ade-addc-c7b180433b35-istio-csr-dns-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:34:18.664354 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.664316 5300 generic.go:356] "Generic (PLEG): container finished" podID="c8404a58-57e8-4ade-addc-c7b180433b35" containerID="39fe6b535c6fbb35041bd132ddf410a95abb5ec5760ee9345bc23ffd4607d546" exitCode=0 Oct 30 01:34:18.664354 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.664359 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" Oct 30 01:34:18.664571 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.664373 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" event={"ID":"c8404a58-57e8-4ade-addc-c7b180433b35","Type":"ContainerDied","Data":"39fe6b535c6fbb35041bd132ddf410a95abb5ec5760ee9345bc23ffd4607d546"} Oct 30 01:34:18.664571 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.664422 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj" event={"ID":"c8404a58-57e8-4ade-addc-c7b180433b35","Type":"ContainerDied","Data":"532079d2e012a0b45133bdb527a19fb9cda6686a75092a10550ad95075fe95f1"} Oct 30 01:34:18.664571 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.664447 5300 scope.go:122] "RemoveContainer" containerID="39fe6b535c6fbb35041bd132ddf410a95abb5ec5760ee9345bc23ffd4607d546" Oct 30 01:34:18.674130 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.674109 5300 scope.go:122] "RemoveContainer" containerID="39fe6b535c6fbb35041bd132ddf410a95abb5ec5760ee9345bc23ffd4607d546" Oct 30 01:34:18.674376 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:34:18.674353 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39fe6b535c6fbb35041bd132ddf410a95abb5ec5760ee9345bc23ffd4607d546\": container with ID starting with 39fe6b535c6fbb35041bd132ddf410a95abb5ec5760ee9345bc23ffd4607d546 not found: ID does not exist" containerID="39fe6b535c6fbb35041bd132ddf410a95abb5ec5760ee9345bc23ffd4607d546" Oct 30 01:34:18.674434 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.674385 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39fe6b535c6fbb35041bd132ddf410a95abb5ec5760ee9345bc23ffd4607d546"} err="failed to get container status \"39fe6b535c6fbb35041bd132ddf410a95abb5ec5760ee9345bc23ffd4607d546\": rpc error: code = NotFound desc = could not find container \"39fe6b535c6fbb35041bd132ddf410a95abb5ec5760ee9345bc23ffd4607d546\": container with ID starting with 39fe6b535c6fbb35041bd132ddf410a95abb5ec5760ee9345bc23ffd4607d546 not found: ID does not exist" Oct 30 01:34:18.720918 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.720892 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj"] Oct 30 01:34:18.726176 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:18.726129 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ppsvj"] Oct 30 01:34:19.820162 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:19.820130 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8404a58-57e8-4ade-addc-c7b180433b35" path="/var/lib/kubelet/pods/c8404a58-57e8-4ade-addc-c7b180433b35/volumes" Oct 30 01:34:21.591120 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:34:21.591081 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:34:28.837458 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837427 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2"] Oct 30 01:34:28.837889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837739 5300 cpu_manager.go:414] "containerMap: removing container" podUID="789a14f0-e4ff-4e2e-b587-3ed5cca21d54" containerName="cleanup" Oct 30 01:34:28.837889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837748 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789a14f0-e4ff-4e2e-b587-3ed5cca21d54" containerName="cleanup" Oct 30 01:34:28.837889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837775 5300 cpu_manager.go:414] "containerMap: removing container" podUID="78aef360-fd10-4741-b251-30ab37a63a05" containerName="payload-pre-processing" Oct 30 01:34:28.837889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837781 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78aef360-fd10-4741-b251-30ab37a63a05" containerName="payload-pre-processing" Oct 30 01:34:28.837889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837795 5300 cpu_manager.go:414] "containerMap: removing container" podUID="ec2520e7-a5f7-4127-ac4b-5bb40efc6309" containerName="maas-api" Oct 30 01:34:28.837889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837800 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2520e7-a5f7-4127-ac4b-5bb40efc6309" containerName="maas-api" Oct 30 01:34:28.837889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837807 5300 cpu_manager.go:414] "containerMap: removing container" podUID="72cf531e-9c97-4a15-a1a4-0beb72c53371" containerName="istio-proxy" Oct 30 01:34:28.837889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837812 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72cf531e-9c97-4a15-a1a4-0beb72c53371" containerName="istio-proxy" Oct 30 01:34:28.837889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837818 5300 cpu_manager.go:414] "containerMap: removing container" podUID="a8d5304a-f4ce-472e-b78f-e5d394677bcd" containerName="revoke-keys" Oct 30 01:34:28.837889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837822 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8d5304a-f4ce-472e-b78f-e5d394677bcd" containerName="revoke-keys" Oct 30 01:34:28.837889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837831 5300 cpu_manager.go:414] "containerMap: removing container" podUID="cfadd1c5-3001-4ebb-8156-46628aa2e722" containerName="istio-proxy" Oct 30 01:34:28.837889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837835 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfadd1c5-3001-4ebb-8156-46628aa2e722" containerName="istio-proxy" Oct 30 01:34:28.837889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837841 5300 cpu_manager.go:414] "containerMap: removing container" podUID="e15f0895-ce3b-401d-af86-68670b59de95" containerName="cleanup" Oct 30 01:34:28.837889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837846 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e15f0895-ce3b-401d-af86-68670b59de95" containerName="cleanup" Oct 30 01:34:28.837889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837851 5300 cpu_manager.go:414] "containerMap: removing container" podUID="2c39939a-6ce7-4527-b066-ce41e01d74d4" containerName="cleanup" Oct 30 01:34:28.837889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837855 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c39939a-6ce7-4527-b066-ce41e01d74d4" containerName="cleanup" Oct 30 01:34:28.837889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837863 5300 cpu_manager.go:414] "containerMap: removing container" podUID="2c39939a-6ce7-4527-b066-ce41e01d74d4" containerName="cleanup" Oct 30 01:34:28.837889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837868 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c39939a-6ce7-4527-b066-ce41e01d74d4" containerName="cleanup" Oct 30 01:34:28.837889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837873 5300 cpu_manager.go:414] "containerMap: removing container" podUID="0577cfbe-a5b1-4408-af05-ec051233cb19" containerName="revoke-keys" Oct 30 01:34:28.837889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837878 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0577cfbe-a5b1-4408-af05-ec051233cb19" containerName="revoke-keys" Oct 30 01:34:28.837889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837884 5300 cpu_manager.go:414] "containerMap: removing container" podUID="72cf531e-9c97-4a15-a1a4-0beb72c53371" containerName="istio-proxy" Oct 30 01:34:28.837889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837888 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72cf531e-9c97-4a15-a1a4-0beb72c53371" containerName="istio-proxy" Oct 30 01:34:28.837889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837903 5300 cpu_manager.go:414] "containerMap: removing container" podUID="789a14f0-e4ff-4e2e-b587-3ed5cca21d54" containerName="cleanup" Oct 30 01:34:28.837889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837908 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789a14f0-e4ff-4e2e-b587-3ed5cca21d54" containerName="cleanup" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837915 5300 cpu_manager.go:414] "containerMap: removing container" podUID="c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a" containerName="cleanup" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837920 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a" containerName="cleanup" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837929 5300 cpu_manager.go:414] "containerMap: removing container" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837933 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837939 5300 cpu_manager.go:414] "containerMap: removing container" podUID="ba5bee00-9a0a-4a43-abea-35da290cbcaa" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837944 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba5bee00-9a0a-4a43-abea-35da290cbcaa" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837949 5300 cpu_manager.go:414] "containerMap: removing container" podUID="78eeb03d-9184-4a3d-af85-b8391731e828" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837954 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78eeb03d-9184-4a3d-af85-b8391731e828" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837958 5300 cpu_manager.go:414] "containerMap: removing container" podUID="508889de-4033-41dd-9b1a-88925f799310" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837962 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="508889de-4033-41dd-9b1a-88925f799310" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837968 5300 cpu_manager.go:414] "containerMap: removing container" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerName="main" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837972 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerName="main" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837978 5300 cpu_manager.go:414] "containerMap: removing container" podUID="f79971c3-5d7b-4fd1-a84f-46b2c0190af6" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837981 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f79971c3-5d7b-4fd1-a84f-46b2c0190af6" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837987 5300 cpu_manager.go:414] "containerMap: removing container" podUID="1e2be936-3c18-4d9b-87a4-800cb91748b7" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837992 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e2be936-3c18-4d9b-87a4-800cb91748b7" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.837999 5300 cpu_manager.go:414] "containerMap: removing container" podUID="8fd0d5d9-adcd-4d24-8916-379be6b844e9" containerName="payload-processing" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838004 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fd0d5d9-adcd-4d24-8916-379be6b844e9" containerName="payload-processing" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838010 5300 cpu_manager.go:414] "containerMap: removing container" podUID="789a14f0-e4ff-4e2e-b587-3ed5cca21d54" containerName="cleanup" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838014 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789a14f0-e4ff-4e2e-b587-3ed5cca21d54" containerName="cleanup" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838020 5300 cpu_manager.go:414] "containerMap: removing container" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerName="main" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838024 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerName="main" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838032 5300 cpu_manager.go:414] "containerMap: removing container" podUID="12e75f96-d218-400a-8b56-0cea344f3e41" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838036 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12e75f96-d218-400a-8b56-0cea344f3e41" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838041 5300 cpu_manager.go:414] "containerMap: removing container" podUID="c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a" containerName="cleanup" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838046 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a" containerName="cleanup" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838051 5300 cpu_manager.go:414] "containerMap: removing container" podUID="e15f0895-ce3b-401d-af86-68670b59de95" containerName="cleanup" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838055 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e15f0895-ce3b-401d-af86-68670b59de95" containerName="cleanup" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838061 5300 cpu_manager.go:414] "containerMap: removing container" podUID="31f807be-26ff-4f56-9e5b-626f349d6206" containerName="payload-processing" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838066 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31f807be-26ff-4f56-9e5b-626f349d6206" containerName="payload-processing" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838072 5300 cpu_manager.go:414] "containerMap: removing container" podUID="2c39939a-6ce7-4527-b066-ce41e01d74d4" containerName="cleanup" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838076 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c39939a-6ce7-4527-b066-ce41e01d74d4" containerName="cleanup" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838083 5300 cpu_manager.go:414] "containerMap: removing container" podUID="19df0d00-62d5-4c6c-9cba-673cd02dbb8a" containerName="payload-pre-processing" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838091 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19df0d00-62d5-4c6c-9cba-673cd02dbb8a" containerName="payload-pre-processing" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838097 5300 cpu_manager.go:414] "containerMap: removing container" podUID="27a25c57-e305-43ad-9de6-e32231466d85" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838102 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27a25c57-e305-43ad-9de6-e32231466d85" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838108 5300 cpu_manager.go:414] "containerMap: removing container" podUID="85d29953-787a-49a2-87d5-1514f1c425ee" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838112 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85d29953-787a-49a2-87d5-1514f1c425ee" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838117 5300 cpu_manager.go:414] "containerMap: removing container" podUID="e15f0895-ce3b-401d-af86-68670b59de95" containerName="cleanup" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838121 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e15f0895-ce3b-401d-af86-68670b59de95" containerName="cleanup" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838128 5300 cpu_manager.go:414] "containerMap: removing container" podUID="e7d2e040-342a-4ea4-a23d-c36722f2d64e" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838132 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7d2e040-342a-4ea4-a23d-c36722f2d64e" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838140 5300 cpu_manager.go:414] "containerMap: removing container" podUID="c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a" containerName="cleanup" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838144 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a" containerName="cleanup" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838151 5300 cpu_manager.go:414] "containerMap: removing container" podUID="7f309af2-1f23-4295-8187-129275259d8a" containerName="payload-processing" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838155 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f309af2-1f23-4295-8187-129275259d8a" containerName="payload-processing" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838160 5300 cpu_manager.go:414] "containerMap: removing container" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838164 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838170 5300 cpu_manager.go:414] "containerMap: removing container" podUID="24445fd3-1ef0-4523-9548-7d1400bcb32b" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838174 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24445fd3-1ef0-4523-9548-7d1400bcb32b" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838179 5300 cpu_manager.go:414] "containerMap: removing container" podUID="aa7f5966-fe14-4c2c-af88-ac82100209ba" containerName="payload-processing" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838183 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa7f5966-fe14-4c2c-af88-ac82100209ba" containerName="payload-processing" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838188 5300 cpu_manager.go:414] "containerMap: removing container" podUID="016eaf4d-095e-4eec-9cae-2c3cb12e13b6" containerName="maas-api" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838192 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="016eaf4d-095e-4eec-9cae-2c3cb12e13b6" containerName="maas-api" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838198 5300 cpu_manager.go:414] "containerMap: removing container" podUID="c8404a58-57e8-4ade-addc-c7b180433b35" containerName="discovery" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838201 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8404a58-57e8-4ade-addc-c7b180433b35" containerName="discovery" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838209 5300 cpu_manager.go:414] "containerMap: removing container" podUID="76bdf51b-ab36-40c2-9284-6cd0e71f1457" containerName="manager" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838212 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76bdf51b-ab36-40c2-9284-6cd0e71f1457" containerName="manager" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838218 5300 cpu_manager.go:414] "containerMap: removing container" podUID="c2839a72-aea8-412d-b105-cf819e2e497a" containerName="maas-api" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838222 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2839a72-aea8-412d-b105-cf819e2e497a" containerName="maas-api" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838228 5300 cpu_manager.go:414] "containerMap: removing container" podUID="db62d92e-f4b6-480b-9545-569baf9670a8" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838232 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db62d92e-f4b6-480b-9545-569baf9670a8" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838239 5300 cpu_manager.go:414] "containerMap: removing container" podUID="459d912e-ed51-4c50-a747-3a2767b30ee2" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838245 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="459d912e-ed51-4c50-a747-3a2767b30ee2" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838252 5300 cpu_manager.go:414] "containerMap: removing container" podUID="5c19f786-50e8-4dee-97ae-c4445035f930" containerName="payload-pre-processing" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838258 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c19f786-50e8-4dee-97ae-c4445035f930" containerName="payload-pre-processing" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838264 5300 cpu_manager.go:414] "containerMap: removing container" podUID="fdd0a075-d5cf-4982-9623-c5432a308de6" containerName="maas-api" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838268 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdd0a075-d5cf-4982-9623-c5432a308de6" containerName="maas-api" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838274 5300 cpu_manager.go:414] "containerMap: removing container" podUID="a2c3a225-7324-4958-8533-29b9c9d2919b" containerName="maas-api" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838278 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2c3a225-7324-4958-8533-29b9c9d2919b" containerName="maas-api" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838329 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="76bdf51b-ab36-40c2-9284-6cd0e71f1457" containerName="manager" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838338 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="78eeb03d-9184-4a3d-af85-b8391731e828" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838344 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2839a72-aea8-412d-b105-cf819e2e497a" containerName="maas-api" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838349 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="db62d92e-f4b6-480b-9545-569baf9670a8" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838356 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="459d912e-ed51-4c50-a747-3a2767b30ee2" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838362 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c19f786-50e8-4dee-97ae-c4445035f930" containerName="payload-pre-processing" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838367 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdd0a075-d5cf-4982-9623-c5432a308de6" containerName="maas-api" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838373 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2c3a225-7324-4958-8533-29b9c9d2919b" containerName="maas-api" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838378 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="789a14f0-e4ff-4e2e-b587-3ed5cca21d54" containerName="cleanup" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838383 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="78aef360-fd10-4741-b251-30ab37a63a05" containerName="payload-pre-processing" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838387 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2520e7-a5f7-4127-ac4b-5bb40efc6309" containerName="maas-api" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838393 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="72cf531e-9c97-4a15-a1a4-0beb72c53371" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838398 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8d5304a-f4ce-472e-b78f-e5d394677bcd" containerName="revoke-keys" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838402 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfadd1c5-3001-4ebb-8156-46628aa2e722" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838407 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="e15f0895-ce3b-401d-af86-68670b59de95" containerName="cleanup" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838411 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c39939a-6ce7-4527-b066-ce41e01d74d4" containerName="cleanup" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838416 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c39939a-6ce7-4527-b066-ce41e01d74d4" containerName="cleanup" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838421 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="0577cfbe-a5b1-4408-af05-ec051233cb19" containerName="revoke-keys" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838426 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="72cf531e-9c97-4a15-a1a4-0beb72c53371" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838431 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="789a14f0-e4ff-4e2e-b587-3ed5cca21d54" containerName="cleanup" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838435 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a" containerName="cleanup" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838440 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="e51b311d-e368-4ecc-80a7-d1cb4a4ed8b7" containerName="main" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838444 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba5bee00-9a0a-4a43-abea-35da290cbcaa" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838449 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="508889de-4033-41dd-9b1a-88925f799310" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838454 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="78eeb03d-9184-4a3d-af85-b8391731e828" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838459 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="f79971c3-5d7b-4fd1-a84f-46b2c0190af6" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838464 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="a36b0810-b564-4c4f-900f-6ec73fabd405" containerName="main" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838470 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e2be936-3c18-4d9b-87a4-800cb91748b7" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838476 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fd0d5d9-adcd-4d24-8916-379be6b844e9" containerName="payload-processing" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838480 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="789a14f0-e4ff-4e2e-b587-3ed5cca21d54" containerName="cleanup" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838485 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d4a7520-7709-4747-b6b1-dda7c86290c1" containerName="main" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838490 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="12e75f96-d218-400a-8b56-0cea344f3e41" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838495 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="e15f0895-ce3b-401d-af86-68670b59de95" containerName="cleanup" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838500 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="31f807be-26ff-4f56-9e5b-626f349d6206" containerName="payload-processing" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838506 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="19df0d00-62d5-4c6c-9cba-673cd02dbb8a" containerName="payload-pre-processing" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838511 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="27a25c57-e305-43ad-9de6-e32231466d85" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838515 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="85d29953-787a-49a2-87d5-1514f1c425ee" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838519 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="e15f0895-ce3b-401d-af86-68670b59de95" containerName="cleanup" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838524 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7d2e040-342a-4ea4-a23d-c36722f2d64e" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838528 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a" containerName="cleanup" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838533 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f309af2-1f23-4295-8187-129275259d8a" containerName="payload-processing" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838537 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="da1ea263-175e-4b07-b74d-a30b16d9ed8c" containerName="main" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838541 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="85d29953-787a-49a2-87d5-1514f1c425ee" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838546 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="24445fd3-1ef0-4523-9548-7d1400bcb32b" containerName="istio-proxy" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838551 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa7f5966-fe14-4c2c-af88-ac82100209ba" containerName="payload-processing" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838556 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="016eaf4d-095e-4eec-9cae-2c3cb12e13b6" containerName="maas-api" Oct 30 01:34:28.838749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.838560 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8404a58-57e8-4ade-addc-c7b180433b35" containerName="discovery" Oct 30 01:34:28.842878 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.842862 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:28.864945 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.864729 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2"] Oct 30 01:34:28.874337 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.874306 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w"] Oct 30 01:34:28.874922 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.874729 5300 cpu_manager.go:414] "containerMap: removing container" podUID="78eeb03d-9184-4a3d-af85-b8391731e828" containerName="istio-proxy" Oct 30 01:34:28.874922 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.874750 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78eeb03d-9184-4a3d-af85-b8391731e828" containerName="istio-proxy" Oct 30 01:34:28.874922 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.874791 5300 cpu_manager.go:414] "containerMap: removing container" podUID="85d29953-787a-49a2-87d5-1514f1c425ee" containerName="istio-proxy" Oct 30 01:34:28.874922 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.874799 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85d29953-787a-49a2-87d5-1514f1c425ee" containerName="istio-proxy" Oct 30 01:34:28.874922 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.874880 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="c097aa62-d0cc-4f5d-9d98-c5a9133f4d3a" containerName="cleanup" Oct 30 01:34:28.874922 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.874895 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c39939a-6ce7-4527-b066-ce41e01d74d4" containerName="cleanup" Oct 30 01:34:28.878013 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.877991 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:28.898024 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.897993 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w"] Oct 30 01:34:28.991720 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.991691 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eea55cea-9b44-45d9-b25f-0b9ca254de04-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-scn6w\" (UID: \"eea55cea-9b44-45d9-b25f-0b9ca254de04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:28.991889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.991731 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eea55cea-9b44-45d9-b25f-0b9ca254de04-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-scn6w\" (UID: \"eea55cea-9b44-45d9-b25f-0b9ca254de04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:28.991889 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.991827 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f16e1d3-330a-4e81-b059-107bf40670b7-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-dzvm2\" (UID: \"5f16e1d3-330a-4e81-b059-107bf40670b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:28.991964 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.991908 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f16e1d3-330a-4e81-b059-107bf40670b7-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dzvm2\" (UID: \"5f16e1d3-330a-4e81-b059-107bf40670b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:28.992006 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.991956 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f16e1d3-330a-4e81-b059-107bf40670b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-dzvm2\" (UID: \"5f16e1d3-330a-4e81-b059-107bf40670b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:28.992044 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.992002 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eea55cea-9b44-45d9-b25f-0b9ca254de04-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-scn6w\" (UID: \"eea55cea-9b44-45d9-b25f-0b9ca254de04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:28.992091 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.992075 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea55cea-9b44-45d9-b25f-0b9ca254de04-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-scn6w\" (UID: \"eea55cea-9b44-45d9-b25f-0b9ca254de04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:28.992128 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.992108 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f16e1d3-330a-4e81-b059-107bf40670b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-dzvm2\" (UID: \"5f16e1d3-330a-4e81-b059-107bf40670b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:28.992165 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.992127 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvgz7\" (UniqueName: \"kubernetes.io/projected/eea55cea-9b44-45d9-b25f-0b9ca254de04-kube-api-access-mvgz7\") pod \"maas-default-gateway-openshift-default-67bf986f99-scn6w\" (UID: \"eea55cea-9b44-45d9-b25f-0b9ca254de04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:28.992200 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.992185 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f16e1d3-330a-4e81-b059-107bf40670b7-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dzvm2\" (UID: \"5f16e1d3-330a-4e81-b059-107bf40670b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:28.992236 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.992223 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea55cea-9b44-45d9-b25f-0b9ca254de04-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-scn6w\" (UID: \"eea55cea-9b44-45d9-b25f-0b9ca254de04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:28.992270 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.992254 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eea55cea-9b44-45d9-b25f-0b9ca254de04-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-scn6w\" (UID: \"eea55cea-9b44-45d9-b25f-0b9ca254de04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:28.992311 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.992278 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f16e1d3-330a-4e81-b059-107bf40670b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-dzvm2\" (UID: \"5f16e1d3-330a-4e81-b059-107bf40670b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:28.992396 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.992383 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f16e1d3-330a-4e81-b059-107bf40670b7-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-dzvm2\" (UID: \"5f16e1d3-330a-4e81-b059-107bf40670b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:28.992440 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.992407 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f16e1d3-330a-4e81-b059-107bf40670b7-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-dzvm2\" (UID: \"5f16e1d3-330a-4e81-b059-107bf40670b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:28.992440 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.992424 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqmzl\" (UniqueName: \"kubernetes.io/projected/5f16e1d3-330a-4e81-b059-107bf40670b7-kube-api-access-jqmzl\") pod \"maas-default-gateway-openshift-default-67bf986f99-dzvm2\" (UID: \"5f16e1d3-330a-4e81-b059-107bf40670b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:28.992546 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.992529 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eea55cea-9b44-45d9-b25f-0b9ca254de04-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-scn6w\" (UID: \"eea55cea-9b44-45d9-b25f-0b9ca254de04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:28.992583 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:28.992561 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eea55cea-9b44-45d9-b25f-0b9ca254de04-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-scn6w\" (UID: \"eea55cea-9b44-45d9-b25f-0b9ca254de04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:29.093815 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.093738 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f16e1d3-330a-4e81-b059-107bf40670b7-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-dzvm2\" (UID: \"5f16e1d3-330a-4e81-b059-107bf40670b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:29.093815 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.093787 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f16e1d3-330a-4e81-b059-107bf40670b7-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-dzvm2\" (UID: \"5f16e1d3-330a-4e81-b059-107bf40670b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:29.093986 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.093903 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmzl\" (UniqueName: \"kubernetes.io/projected/5f16e1d3-330a-4e81-b059-107bf40670b7-kube-api-access-jqmzl\") pod \"maas-default-gateway-openshift-default-67bf986f99-dzvm2\" (UID: \"5f16e1d3-330a-4e81-b059-107bf40670b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:29.093986 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.093936 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eea55cea-9b44-45d9-b25f-0b9ca254de04-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-scn6w\" (UID: \"eea55cea-9b44-45d9-b25f-0b9ca254de04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:29.093986 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.093964 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eea55cea-9b44-45d9-b25f-0b9ca254de04-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-scn6w\" (UID: \"eea55cea-9b44-45d9-b25f-0b9ca254de04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:29.094172 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.094003 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eea55cea-9b44-45d9-b25f-0b9ca254de04-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-scn6w\" (UID: \"eea55cea-9b44-45d9-b25f-0b9ca254de04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:29.094172 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.094043 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eea55cea-9b44-45d9-b25f-0b9ca254de04-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-scn6w\" (UID: \"eea55cea-9b44-45d9-b25f-0b9ca254de04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:29.094172 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.094076 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f16e1d3-330a-4e81-b059-107bf40670b7-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-dzvm2\" (UID: \"5f16e1d3-330a-4e81-b059-107bf40670b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:29.094172 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.094096 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f16e1d3-330a-4e81-b059-107bf40670b7-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-dzvm2\" (UID: \"5f16e1d3-330a-4e81-b059-107bf40670b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:29.094172 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.094102 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f16e1d3-330a-4e81-b059-107bf40670b7-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dzvm2\" (UID: \"5f16e1d3-330a-4e81-b059-107bf40670b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:29.094172 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.094164 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f16e1d3-330a-4e81-b059-107bf40670b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-dzvm2\" (UID: \"5f16e1d3-330a-4e81-b059-107bf40670b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:29.094505 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.094189 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eea55cea-9b44-45d9-b25f-0b9ca254de04-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-scn6w\" (UID: \"eea55cea-9b44-45d9-b25f-0b9ca254de04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:29.094505 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.094229 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea55cea-9b44-45d9-b25f-0b9ca254de04-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-scn6w\" (UID: \"eea55cea-9b44-45d9-b25f-0b9ca254de04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:29.094505 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.094266 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f16e1d3-330a-4e81-b059-107bf40670b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-dzvm2\" (UID: \"5f16e1d3-330a-4e81-b059-107bf40670b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:29.094505 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.094293 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvgz7\" (UniqueName: \"kubernetes.io/projected/eea55cea-9b44-45d9-b25f-0b9ca254de04-kube-api-access-mvgz7\") pod \"maas-default-gateway-openshift-default-67bf986f99-scn6w\" (UID: \"eea55cea-9b44-45d9-b25f-0b9ca254de04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:29.094505 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.094346 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f16e1d3-330a-4e81-b059-107bf40670b7-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dzvm2\" (UID: \"5f16e1d3-330a-4e81-b059-107bf40670b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:29.094505 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.094386 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea55cea-9b44-45d9-b25f-0b9ca254de04-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-scn6w\" (UID: \"eea55cea-9b44-45d9-b25f-0b9ca254de04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:29.094505 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.094397 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f16e1d3-330a-4e81-b059-107bf40670b7-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dzvm2\" (UID: \"5f16e1d3-330a-4e81-b059-107bf40670b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:29.094505 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.094425 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eea55cea-9b44-45d9-b25f-0b9ca254de04-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-scn6w\" (UID: \"eea55cea-9b44-45d9-b25f-0b9ca254de04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:29.094505 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.094478 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eea55cea-9b44-45d9-b25f-0b9ca254de04-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-scn6w\" (UID: \"eea55cea-9b44-45d9-b25f-0b9ca254de04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:29.095032 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.094515 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f16e1d3-330a-4e81-b059-107bf40670b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-dzvm2\" (UID: \"5f16e1d3-330a-4e81-b059-107bf40670b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:29.095032 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.094629 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea55cea-9b44-45d9-b25f-0b9ca254de04-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-scn6w\" (UID: \"eea55cea-9b44-45d9-b25f-0b9ca254de04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:29.095032 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.094710 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f16e1d3-330a-4e81-b059-107bf40670b7-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dzvm2\" (UID: \"5f16e1d3-330a-4e81-b059-107bf40670b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:29.095032 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.094795 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eea55cea-9b44-45d9-b25f-0b9ca254de04-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-scn6w\" (UID: \"eea55cea-9b44-45d9-b25f-0b9ca254de04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:29.095032 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.094850 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f16e1d3-330a-4e81-b059-107bf40670b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-dzvm2\" (UID: \"5f16e1d3-330a-4e81-b059-107bf40670b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:29.095032 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.094892 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea55cea-9b44-45d9-b25f-0b9ca254de04-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-scn6w\" (UID: \"eea55cea-9b44-45d9-b25f-0b9ca254de04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:29.095359 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.095090 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f16e1d3-330a-4e81-b059-107bf40670b7-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-dzvm2\" (UID: \"5f16e1d3-330a-4e81-b059-107bf40670b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:29.095359 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.095117 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eea55cea-9b44-45d9-b25f-0b9ca254de04-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-scn6w\" (UID: \"eea55cea-9b44-45d9-b25f-0b9ca254de04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:29.096651 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.096629 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eea55cea-9b44-45d9-b25f-0b9ca254de04-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-scn6w\" (UID: \"eea55cea-9b44-45d9-b25f-0b9ca254de04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:29.096794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.096723 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eea55cea-9b44-45d9-b25f-0b9ca254de04-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-scn6w\" (UID: \"eea55cea-9b44-45d9-b25f-0b9ca254de04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:29.096917 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.096900 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f16e1d3-330a-4e81-b059-107bf40670b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-dzvm2\" (UID: \"5f16e1d3-330a-4e81-b059-107bf40670b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:29.097144 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.097127 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f16e1d3-330a-4e81-b059-107bf40670b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-dzvm2\" (UID: \"5f16e1d3-330a-4e81-b059-107bf40670b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:29.115183 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.115157 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f16e1d3-330a-4e81-b059-107bf40670b7-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-dzvm2\" (UID: \"5f16e1d3-330a-4e81-b059-107bf40670b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:29.115559 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.115533 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvgz7\" (UniqueName: \"kubernetes.io/projected/eea55cea-9b44-45d9-b25f-0b9ca254de04-kube-api-access-mvgz7\") pod \"maas-default-gateway-openshift-default-67bf986f99-scn6w\" (UID: \"eea55cea-9b44-45d9-b25f-0b9ca254de04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:29.115607 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.115544 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eea55cea-9b44-45d9-b25f-0b9ca254de04-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-scn6w\" (UID: \"eea55cea-9b44-45d9-b25f-0b9ca254de04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:29.115683 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.115668 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqmzl\" (UniqueName: \"kubernetes.io/projected/5f16e1d3-330a-4e81-b059-107bf40670b7-kube-api-access-jqmzl\") pod \"maas-default-gateway-openshift-default-67bf986f99-dzvm2\" (UID: \"5f16e1d3-330a-4e81-b059-107bf40670b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:29.155421 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.155400 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:29.206180 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.205951 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:29.465500 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.465475 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2"] Oct 30 01:34:29.466650 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:34:29.466624 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f16e1d3_330a_4e81_b059_107bf40670b7.slice/crio-6a0a8ffe3991ff7a4640d21826ee98d4ebf7005d9eb54452a177e50720262a68 WatchSource:0}: Error finding container 6a0a8ffe3991ff7a4640d21826ee98d4ebf7005d9eb54452a177e50720262a68: Status 404 returned error can't find the container with id 6a0a8ffe3991ff7a4640d21826ee98d4ebf7005d9eb54452a177e50720262a68 Oct 30 01:34:29.468770 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.468723 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:34:29.468882 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.468817 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:34:29.468882 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.468863 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:34:29.511411 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.511383 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w"] Oct 30 01:34:29.512350 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:34:29.512327 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea55cea_9b44_45d9_b25f_0b9ca254de04.slice/crio-3fbba132f5e4b8c22111992778d67cf4b6c1dcd86389f83e5998ebbb371ba1dc WatchSource:0}: Error finding container 3fbba132f5e4b8c22111992778d67cf4b6c1dcd86389f83e5998ebbb371ba1dc: Status 404 returned error can't find the container with id 3fbba132f5e4b8c22111992778d67cf4b6c1dcd86389f83e5998ebbb371ba1dc Oct 30 01:34:29.514486 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.514453 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:34:29.514575 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.514516 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:34:29.514575 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.514545 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:34:29.713928 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.713817 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" event={"ID":"eea55cea-9b44-45d9-b25f-0b9ca254de04","Type":"ContainerStarted","Data":"6f1fe15a7b572408ba0494ef77546dd9b7c3b9b5e94c557bbc3c08a80739798d"} Oct 30 01:34:29.713928 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.713897 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" event={"ID":"eea55cea-9b44-45d9-b25f-0b9ca254de04","Type":"ContainerStarted","Data":"3fbba132f5e4b8c22111992778d67cf4b6c1dcd86389f83e5998ebbb371ba1dc"} Oct 30 01:34:29.715564 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.715540 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" event={"ID":"5f16e1d3-330a-4e81-b059-107bf40670b7","Type":"ContainerStarted","Data":"751e2f510ef9469f9099b6301b4f06b0cbbfd803f94db44e327b27be2652f79b"} Oct 30 01:34:29.715686 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.715587 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" event={"ID":"5f16e1d3-330a-4e81-b059-107bf40670b7","Type":"ContainerStarted","Data":"6a0a8ffe3991ff7a4640d21826ee98d4ebf7005d9eb54452a177e50720262a68"} Oct 30 01:34:29.760961 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.760882 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" podStartSLOduration=1.760863952 podStartE2EDuration="1.760863952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:29.759131285 +0000 UTC m=+3070.074999634" watchObservedRunningTime="2026-10-30 01:34:29.760863952 +0000 UTC m=+3070.076732279" Oct 30 01:34:29.808506 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:29.808449 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" podStartSLOduration=1.808430787 podStartE2EDuration="1.808430787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:29.806508289 +0000 UTC m=+3070.122376613" watchObservedRunningTime="2026-10-30 01:34:29.808430787 +0000 UTC m=+3070.124299111" Oct 30 01:34:30.158184 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:30.156150 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:30.158184 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:30.157823 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dzvm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 01:34:30.158184 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:30.157875 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" podUID="5f16e1d3-330a-4e81-b059-107bf40670b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 01:34:30.207015 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:30.206976 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:30.208323 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:30.208301 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-scn6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 01:34:30.208397 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:30.208349 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" podUID="eea55cea-9b44-45d9-b25f-0b9ca254de04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 01:34:31.156713 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:31.156190 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dzvm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 01:34:31.156713 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:31.156256 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" podUID="5f16e1d3-330a-4e81-b059-107bf40670b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 01:34:31.207608 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:31.207568 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-scn6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 01:34:31.208015 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:31.207632 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" podUID="eea55cea-9b44-45d9-b25f-0b9ca254de04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 01:34:32.169081 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:32.168860 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dzvm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 01:34:32.169081 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:32.168919 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" podUID="5f16e1d3-330a-4e81-b059-107bf40670b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 01:34:32.207583 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:32.207554 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-scn6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 01:34:32.207756 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:32.207603 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" podUID="eea55cea-9b44-45d9-b25f-0b9ca254de04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 01:34:33.156224 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:33.156185 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dzvm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 01:34:33.156519 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:33.156491 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" podUID="5f16e1d3-330a-4e81-b059-107bf40670b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 01:34:33.206712 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:33.206684 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-scn6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 01:34:33.206874 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:33.206733 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" podUID="eea55cea-9b44-45d9-b25f-0b9ca254de04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 01:34:34.168051 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:34.168010 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dzvm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 01:34:34.168651 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:34.168607 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" podUID="5f16e1d3-330a-4e81-b059-107bf40670b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 01:34:34.206773 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:34.206732 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-scn6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 01:34:34.206900 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:34.206792 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" podUID="eea55cea-9b44-45d9-b25f-0b9ca254de04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 01:34:35.156218 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:35.156177 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dzvm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 01:34:35.156494 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:35.156467 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" podUID="5f16e1d3-330a-4e81-b059-107bf40670b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 01:34:35.207535 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:35.207500 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-scn6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 01:34:35.208018 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:35.207550 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" podUID="eea55cea-9b44-45d9-b25f-0b9ca254de04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 01:34:35.966078 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:35.966031 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:36.155888 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:36.155845 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dzvm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 01:34:36.156100 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:36.155916 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" podUID="5f16e1d3-330a-4e81-b059-107bf40670b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 01:34:36.207363 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:36.207322 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-scn6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 01:34:36.207567 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:36.207370 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" podUID="eea55cea-9b44-45d9-b25f-0b9ca254de04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 01:34:37.966137 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:37.966105 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:38.156041 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:38.156007 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:38.156996 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:38.156966 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dzvm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:34:38.157082 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:38.157024 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" podUID="5f16e1d3-330a-4e81-b059-107bf40670b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:34:38.208242 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:38.208214 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-scn6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:34:38.208417 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:38.208268 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" podUID="eea55cea-9b44-45d9-b25f-0b9ca254de04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:34:39.197945 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:39.197890 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dzvm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:34:39.198359 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:39.197965 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" podUID="5f16e1d3-330a-4e81-b059-107bf40670b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:34:39.249933 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:39.249885 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-scn6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:34:39.250106 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:39.249947 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" podUID="eea55cea-9b44-45d9-b25f-0b9ca254de04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:34:40.199292 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:40.199254 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dzvm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:34:40.199787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:40.199314 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" podUID="5f16e1d3-330a-4e81-b059-107bf40670b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:34:40.250387 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:40.250343 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-scn6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:34:40.250570 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:40.250405 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" podUID="eea55cea-9b44-45d9-b25f-0b9ca254de04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:34:41.240958 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:41.240910 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dzvm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:34:41.241454 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:41.240980 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" podUID="5f16e1d3-330a-4e81-b059-107bf40670b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:34:41.291932 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:41.291888 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-scn6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:34:41.292089 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:41.291956 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" podUID="eea55cea-9b44-45d9-b25f-0b9ca254de04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:34:42.091629 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:42.091601 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:42.091922 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:42.091903 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:42.092581 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:42.092564 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" Oct 30 01:34:42.332925 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:42.332882 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-scn6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:34:42.333437 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:42.332964 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" podUID="eea55cea-9b44-45d9-b25f-0b9ca254de04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:34:42.667680 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:42.667650 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:42.668015 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:42.667975 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:42.669508 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:42.669486 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-scn6w" Oct 30 01:34:51.592314 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:34:51.592271 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:34:52.001876 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:34:52.001842 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:11.028049 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:35:11.028007 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb26b84d-qgthh"] Oct 30 01:35:11.041716 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:35:11.041687 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb26b84d-qgthh"] Oct 30 01:35:11.849316 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:35:11.849279 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0577cfbe-a5b1-4408-af05-ec051233cb19" path="/var/lib/kubelet/pods/0577cfbe-a5b1-4408-af05-ec051233cb19/volumes" Oct 30 01:35:13.035286 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:35:13.035230 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-389c1c22-52t4q"] Oct 30 01:35:13.049093 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:35:13.049067 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-389c1c22-52t4q"] Oct 30 01:35:13.818602 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:35:13.818561 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8d5304a-f4ce-472e-b78f-e5d394677bcd" path="/var/lib/kubelet/pods/a8d5304a-f4ce-472e-b78f-e5d394677bcd/volumes" Oct 30 01:35:21.590969 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:35:21.590932 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:35:38.750703 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:35:38.750655 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:38.850680 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:35:38.850633 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:51.591141 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:35:51.591104 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:36:21.591091 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:36:21.591004 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:36:21.986129 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:36:21.986102 5300 scope.go:122] "RemoveContainer" containerID="2c59ea7c4af5aaf5a97915ef4247ce2d5de2ed800c8c1b21ba0b4c1df5ba72b5" Oct 30 01:36:21.994020 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:36:21.993996 5300 scope.go:122] "RemoveContainer" containerID="66682d9e2fecb793e1a51c95ee72f32db9644c49781bedd0c07c44a8320c7a0a" Oct 30 01:36:28.138784 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:36:28.137871 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dzvm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:36:28.138784 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:36:28.137943 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" podUID="5f16e1d3-330a-4e81-b059-107bf40670b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:36:36.201789 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:36:36.199543 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:51.592142 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:36:51.592107 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:36:58.133155 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:36:58.132905 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dzvm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:36:58.133155 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:36:58.133001 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" podUID="5f16e1d3-330a-4e81-b059-107bf40670b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:37:03.762929 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:03.762892 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:03.852838 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:03.852797 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:21.592509 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:37:21.592469 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:37:30.676982 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:30.676932 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:36.342467 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:36.342430 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:37.733205 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.733171 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:37.764659 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.764625 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62"] Oct 30 01:37:37.768212 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.768192 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:37.784750 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.784720 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-37ba51-openshift-default-dockercfg-kfp55\"" Oct 30 01:37:37.800970 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.800927 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62"] Oct 30 01:37:37.833614 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.833567 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:37.858965 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.858927 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-token\") pod \"e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:37.859153 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.858977 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-workload-certs\") pod \"e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:37.859153 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.859057 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-podinfo\") pod \"e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:37.859153 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.859115 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istiod-ca-cert\") pod \"e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:37.859266 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.859154 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-envoy\") pod \"e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:37.859266 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.859175 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69zmc\" (UniqueName: \"kubernetes.io/projected/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-kube-api-access-69zmc\") pod \"e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:37.859266 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.859255 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-credential-socket\") pod \"e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:37.859370 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.859281 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-workload-socket\") pod \"e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:37.859370 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.859306 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-data\") pod \"e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:37.960536 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.960501 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-data\") pod \"e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:37.960753 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.960584 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-token\") pod \"e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:37.960753 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.960624 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-workload-certs\") pod \"e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:37.960753 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.960651 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-podinfo\") pod \"e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:37.960753 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.960704 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istiod-ca-cert\") pod \"e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:37.960753 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.960748 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-envoy\") pod \"e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:37.961121 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.960813 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69zmc\" (UniqueName: \"kubernetes.io/projected/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-kube-api-access-69zmc\") pod \"e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:37.961121 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.960853 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-credential-socket\") pod \"e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:37.961121 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.960889 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-workload-socket\") pod \"e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:37.961121 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.961060 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-data\") pod \"e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:37.961121 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.961102 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-workload-certs\") pod \"e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:37.961327 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.961235 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-workload-socket\") pod \"e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:37.961382 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.961357 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-credential-socket\") pod \"e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:37.961617 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.961599 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istiod-ca-cert\") pod \"e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:37.963226 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.963201 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-envoy\") pod \"e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:37.963335 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.963276 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-podinfo\") pod \"e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:37.984729 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.984640 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-token\") pod \"e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:37.984919 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:37.984880 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69zmc\" (UniqueName: \"kubernetes.io/projected/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-kube-api-access-69zmc\") pod \"e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:38.065354 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:38.065318 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:38.081354 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:38.081322 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:38.402074 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:38.402048 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62"] Oct 30 01:37:38.404733 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:37:38.404703 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba721bdc_fdb5_4a6a_ac43_e45f088b7c8f.slice/crio-18b3337114aa202c1acdbbc42acb066c6b13aa77dbee041235147ea745f787a2 WatchSource:0}: Error finding container 18b3337114aa202c1acdbbc42acb066c6b13aa77dbee041235147ea745f787a2: Status 404 returned error can't find the container with id 18b3337114aa202c1acdbbc42acb066c6b13aa77dbee041235147ea745f787a2 Oct 30 01:37:38.406987 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:38.406954 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:37:38.407079 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:38.407023 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:37:38.407079 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:38.407053 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:37:38.734059 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:38.734020 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" event={"ID":"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f","Type":"ContainerStarted","Data":"6ba0c2d8119aa040792b9ef2262c611e84842c38863e1bd05b336b11a12de5ba"} Oct 30 01:37:38.734441 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:38.734089 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" event={"ID":"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f","Type":"ContainerStarted","Data":"18b3337114aa202c1acdbbc42acb066c6b13aa77dbee041235147ea745f787a2"} Oct 30 01:37:38.788890 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:38.788842 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" podStartSLOduration=1.788828368 podStartE2EDuration="1.788828368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:38.788626681 +0000 UTC m=+3259.104495015" watchObservedRunningTime="2026-10-30 01:37:38.788828368 +0000 UTC m=+3259.104696695" Oct 30 01:37:39.081492 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:39.081406 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:39.087524 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:39.087488 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:39.738533 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:39.738499 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:39.739389 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:39.739368 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:41.178386 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:41.178349 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:41.294072 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:41.294040 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:42.776207 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:42.776175 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:42.878954 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:42.878915 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:42.978315 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:42.978280 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:43.170018 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:43.169854 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b"] Oct 30 01:37:43.174116 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:43.174086 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b" Oct 30 01:37:43.207115 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:43.207058 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-6f20578c-dockercfg-xfk8b\"" Oct 30 01:37:43.209407 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:43.209378 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-37ba51\"" Oct 30 01:37:43.210265 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:43.210238 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b"] Oct 30 01:37:43.316173 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:43.316137 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9j8p\" (UniqueName: \"kubernetes.io/projected/21d23e7c-c019-4420-89aa-7468dadb0659-kube-api-access-l9j8p\") pod \"payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b\" (UID: \"21d23e7c-c019-4420-89aa-7468dadb0659\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b" Oct 30 01:37:43.316399 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:43.316204 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d23e7c-c019-4420-89aa-7468dadb0659-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b\" (UID: \"21d23e7c-c019-4420-89aa-7468dadb0659\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b" Oct 30 01:37:43.416840 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:43.416808 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9j8p\" (UniqueName: \"kubernetes.io/projected/21d23e7c-c019-4420-89aa-7468dadb0659-kube-api-access-l9j8p\") pod \"payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b\" (UID: \"21d23e7c-c019-4420-89aa-7468dadb0659\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b" Oct 30 01:37:43.417020 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:43.416877 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d23e7c-c019-4420-89aa-7468dadb0659-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b\" (UID: \"21d23e7c-c019-4420-89aa-7468dadb0659\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b" Oct 30 01:37:43.417596 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:43.417575 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d23e7c-c019-4420-89aa-7468dadb0659-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b\" (UID: \"21d23e7c-c019-4420-89aa-7468dadb0659\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b" Oct 30 01:37:43.443642 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:43.443574 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9j8p\" (UniqueName: \"kubernetes.io/projected/21d23e7c-c019-4420-89aa-7468dadb0659-kube-api-access-l9j8p\") pod \"payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b\" (UID: \"21d23e7c-c019-4420-89aa-7468dadb0659\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b" Oct 30 01:37:43.491650 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:43.491620 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b" Oct 30 01:37:43.789514 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:43.789488 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b"] Oct 30 01:37:43.791987 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:37:43.791959 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d23e7c_c019_4420_89aa_7468dadb0659.slice/crio-de740349eb58925596e54e75a0fbb2e7bce220d1091c03f0b2ee1f5e91900396 WatchSource:0}: Error finding container de740349eb58925596e54e75a0fbb2e7bce220d1091c03f0b2ee1f5e91900396: Status 404 returned error can't find the container with id de740349eb58925596e54e75a0fbb2e7bce220d1091c03f0b2ee1f5e91900396 Oct 30 01:37:44.499776 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:44.499741 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:44.762585 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:44.762486 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b" event={"ID":"21d23e7c-c019-4420-89aa-7468dadb0659","Type":"ContainerStarted","Data":"b07ebdc003d798770a14a0f4b10fc2ae5315d69522937e3e91f019314cdce7ed"} Oct 30 01:37:44.762585 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:44.762547 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b" event={"ID":"21d23e7c-c019-4420-89aa-7468dadb0659","Type":"ContainerStarted","Data":"de740349eb58925596e54e75a0fbb2e7bce220d1091c03f0b2ee1f5e91900396"} Oct 30 01:37:44.762585 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:44.762575 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b" Oct 30 01:37:44.806107 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:44.806058 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b" podStartSLOduration=1.806044945 podStartE2EDuration="1.806044945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:44.805062071 +0000 UTC m=+3265.120930396" watchObservedRunningTime="2026-10-30 01:37:44.806044945 +0000 UTC m=+3265.121913269" Oct 30 01:37:46.261097 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:46.261064 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:47.993817 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:47.993783 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:49.754083 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:49.754043 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:51.414820 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:51.414783 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:51.592645 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:37:51.592609 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:37:53.044074 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:53.044038 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:54.713734 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:54.713698 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:55.770026 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:55.770000 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b" Oct 30 01:37:55.779908 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:55.779885 5300 patch_prober.go:29] interesting pod/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:37:55.779980 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:55.779942 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" podUID="ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:37:56.028710 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:56.028623 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:56.395696 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:56.395670 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:56.672999 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:56.672915 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:56.804843 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:56.804812 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:56.833597 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:56.833569 5300 generic.go:356] "Generic (PLEG): container finished" podID="ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f" containerID="6ba0c2d8119aa040792b9ef2262c611e84842c38863e1bd05b336b11a12de5ba" exitCode=0 Oct 30 01:37:56.833748 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:56.833649 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" event={"ID":"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f","Type":"ContainerDied","Data":"6ba0c2d8119aa040792b9ef2262c611e84842c38863e1bd05b336b11a12de5ba"} Oct 30 01:37:56.834068 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:56.834048 5300 scope.go:122] "RemoveContainer" containerID="6ba0c2d8119aa040792b9ef2262c611e84842c38863e1bd05b336b11a12de5ba" Oct 30 01:37:56.835218 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:56.835185 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:37:56.835325 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:56.835268 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:37:56.835397 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:56.835320 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:37:56.897455 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:56.897430 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:57.850547 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:57.850517 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" event={"ID":"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f","Type":"ContainerStarted","Data":"fed8d78912c1639be4579b2d3a6fb2553269d70dfd94ab8f883d09f404e71e61"} Oct 30 01:37:58.081846 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:58.081811 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:37:58.082171 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:58.082140 5300 patch_prober.go:29] interesting pod/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 30 01:37:58.082246 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:58.082219 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" podUID="ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 30 01:37:58.144278 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:58.144191 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:58.342786 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:58.340996 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:59.082309 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:59.082275 5300 patch_prober.go:29] interesting pod/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 30 01:37:59.082752 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:59.082333 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" podUID="ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 30 01:37:59.998909 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:37:59.998867 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:00.081735 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:00.081702 5300 patch_prober.go:29] interesting pod/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 30 01:38:00.081954 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:00.081789 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" podUID="ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 30 01:38:01.082286 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:01.082234 5300 patch_prober.go:29] interesting pod/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 30 01:38:01.082717 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:01.082325 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" podUID="ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 30 01:38:01.751853 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:01.751815 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:02.081848 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:02.081744 5300 patch_prober.go:29] interesting pod/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 30 01:38:02.081848 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:02.081821 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" podUID="ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 30 01:38:03.081919 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:03.081878 5300 patch_prober.go:29] interesting pod/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 30 01:38:03.082334 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:03.081941 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" podUID="ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 30 01:38:03.516310 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:03.516270 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:04.086077 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:04.086047 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:38:04.086468 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:04.086306 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:38:04.086935 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:04.086917 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:38:05.204396 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:05.204361 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:06.888176 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:06.888138 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:08.488985 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:08.488951 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:10.173097 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:10.173053 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:11.907640 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:11.907609 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:13.700132 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:13.700093 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:15.451942 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:15.451896 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:17.175833 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:17.175794 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:18.732186 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:18.732150 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:20.350084 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:20.350049 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:21.592130 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:38:21.592090 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:38:22.207730 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:22.207697 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:23.810431 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:23.810402 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:25.386464 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:25.386435 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:27.171215 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:27.171183 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:28.748294 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:28.748258 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:30.328737 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:30.328707 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:32.019882 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:32.019851 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:32.652483 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:32.652451 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:33.819453 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:33.819421 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:35.657332 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:35.657301 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:36.550958 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:36.550922 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:37.408537 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:37.408506 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:37.617340 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:37.617307 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:39.006368 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:39.006329 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:40.653962 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:40.653925 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:42.345844 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:42.345748 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:43.214053 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:43.214022 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b"] Oct 30 01:38:43.424047 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:43.424019 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:43.546404 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:43.546355 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:43.834066 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:43.833962 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:46.257806 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:46.257745 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:46.375310 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:46.375278 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:46.841167 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:46.841134 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:48.215278 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:48.215238 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b" podUID="21d23e7c-c019-4420-89aa-7468dadb0659" containerName="payload-processing" containerID="cri-o://b07ebdc003d798770a14a0f4b10fc2ae5315d69522937e3e91f019314cdce7ed" gracePeriod=25 Oct 30 01:38:51.592812 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:38:51.592743 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:38:52.742872 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:52.742831 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:52.853727 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:52.853687 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:53.108895 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:53.108730 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62"] Oct 30 01:38:53.109083 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:53.109058 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" podUID="ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f" containerName="istio-proxy" containerID="cri-o://fed8d78912c1639be4579b2d3a6fb2553269d70dfd94ab8f883d09f404e71e61" gracePeriod=30 Oct 30 01:38:53.131743 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:53.131710 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:55.769204 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:55.769164 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b" podUID="21d23e7c-c019-4420-89aa-7468dadb0659" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:38:58.359961 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.359938 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:38:58.478270 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.478244 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-data\") pod \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " Oct 30 01:38:58.478413 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.478289 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-token\" (UniqueName: \"kubernetes.io/projected/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-token\") pod \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " Oct 30 01:38:58.478413 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.478331 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-podinfo\") pod \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " Oct 30 01:38:58.478413 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.478355 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istiod-ca-cert\") pod \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " Oct 30 01:38:58.478413 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.478397 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-credential-socket\") pod \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " Oct 30 01:38:58.478696 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.478447 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-kube-api-access-69zmc\" (UniqueName: \"kubernetes.io/projected/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-kube-api-access-69zmc\") pod \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " Oct 30 01:38:58.478696 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.478480 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-envoy\") pod \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " Oct 30 01:38:58.478696 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.478565 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-data" pod "ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f" (UID: "ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:58.478696 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.478590 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-workload-certs\") pod \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " Oct 30 01:38:58.478696 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.478623 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-workload-socket\") pod \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\" (UID: \"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f\") " Oct 30 01:38:58.478985 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.478830 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-credential-socket" pod "ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f" (UID: "ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:58.478985 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.478846 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istiod-ca-cert" pod "ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f" (UID: "ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:58.479078 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.478983 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-workload-certs" pod "ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f" (UID: "ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:58.479078 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.479057 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-workload-socket" pod "ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f" (UID: "ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:58.479167 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.479134 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:38:58.479167 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.479156 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:38:58.479261 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.479172 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:38:58.479261 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.479185 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:38:58.479261 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.479197 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:38:58.480770 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.480688 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-podinfo" pod "ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f" (UID: "ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:38:58.480770 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.480740 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-envoy" pod "ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f" (UID: "ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:58.481121 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.481101 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-token" pod "ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f" (UID: "ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:58.481232 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.481216 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-kube-api-access-69zmc" pod "ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f" (UID: "ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f"). InnerVolumeSpecName "kube-api-access-69zmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:58.579972 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.579947 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:38:58.579972 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.579969 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:38:58.580119 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.579978 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69zmc\" (UniqueName: \"kubernetes.io/projected/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-kube-api-access-69zmc\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:38:58.580119 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:58.579988 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:38:59.144824 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:59.144792 5300 generic.go:356] "Generic (PLEG): container finished" podID="ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f" containerID="fed8d78912c1639be4579b2d3a6fb2553269d70dfd94ab8f883d09f404e71e61" exitCode=0 Oct 30 01:38:59.145031 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:59.144846 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" Oct 30 01:38:59.145031 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:59.144848 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" event={"ID":"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f","Type":"ContainerDied","Data":"fed8d78912c1639be4579b2d3a6fb2553269d70dfd94ab8f883d09f404e71e61"} Oct 30 01:38:59.145031 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:59.144990 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62" event={"ID":"ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f","Type":"ContainerDied","Data":"18b3337114aa202c1acdbbc42acb066c6b13aa77dbee041235147ea745f787a2"} Oct 30 01:38:59.145031 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:59.145028 5300 scope.go:122] "RemoveContainer" containerID="fed8d78912c1639be4579b2d3a6fb2553269d70dfd94ab8f883d09f404e71e61" Oct 30 01:38:59.154069 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:59.154053 5300 scope.go:122] "RemoveContainer" containerID="6ba0c2d8119aa040792b9ef2262c611e84842c38863e1bd05b336b11a12de5ba" Oct 30 01:38:59.161716 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:59.161698 5300 scope.go:122] "RemoveContainer" containerID="fed8d78912c1639be4579b2d3a6fb2553269d70dfd94ab8f883d09f404e71e61" Oct 30 01:38:59.162001 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:38:59.161983 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fed8d78912c1639be4579b2d3a6fb2553269d70dfd94ab8f883d09f404e71e61\": container with ID starting with fed8d78912c1639be4579b2d3a6fb2553269d70dfd94ab8f883d09f404e71e61 not found: ID does not exist" containerID="fed8d78912c1639be4579b2d3a6fb2553269d70dfd94ab8f883d09f404e71e61" Oct 30 01:38:59.162085 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:59.162014 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fed8d78912c1639be4579b2d3a6fb2553269d70dfd94ab8f883d09f404e71e61"} err="failed to get container status \"fed8d78912c1639be4579b2d3a6fb2553269d70dfd94ab8f883d09f404e71e61\": rpc error: code = NotFound desc = could not find container \"fed8d78912c1639be4579b2d3a6fb2553269d70dfd94ab8f883d09f404e71e61\": container with ID starting with fed8d78912c1639be4579b2d3a6fb2553269d70dfd94ab8f883d09f404e71e61 not found: ID does not exist" Oct 30 01:38:59.162085 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:59.162039 5300 scope.go:122] "RemoveContainer" containerID="6ba0c2d8119aa040792b9ef2262c611e84842c38863e1bd05b336b11a12de5ba" Oct 30 01:38:59.162306 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:38:59.162288 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ba0c2d8119aa040792b9ef2262c611e84842c38863e1bd05b336b11a12de5ba\": container with ID starting with 6ba0c2d8119aa040792b9ef2262c611e84842c38863e1bd05b336b11a12de5ba not found: ID does not exist" containerID="6ba0c2d8119aa040792b9ef2262c611e84842c38863e1bd05b336b11a12de5ba" Oct 30 01:38:59.162346 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:59.162312 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ba0c2d8119aa040792b9ef2262c611e84842c38863e1bd05b336b11a12de5ba"} err="failed to get container status \"6ba0c2d8119aa040792b9ef2262c611e84842c38863e1bd05b336b11a12de5ba\": rpc error: code = NotFound desc = could not find container \"6ba0c2d8119aa040792b9ef2262c611e84842c38863e1bd05b336b11a12de5ba\": container with ID starting with 6ba0c2d8119aa040792b9ef2262c611e84842c38863e1bd05b336b11a12de5ba not found: ID does not exist" Oct 30 01:38:59.228417 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:59.228384 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62"] Oct 30 01:38:59.242590 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:59.242560 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-37ba51-openshift-default-6d96ccdc97z7k62"] Oct 30 01:38:59.818671 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:38:59.818642 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f" path="/var/lib/kubelet/pods/ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f/volumes" Oct 30 01:39:01.345683 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:39:01.345599 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:01.460730 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:39:01.460700 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:05.768666 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:39:05.768624 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b" podUID="21d23e7c-c019-4420-89aa-7468dadb0659" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:39:08.467196 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:39:08.467170 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b" Oct 30 01:39:08.572936 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:39:08.572848 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21d23e7c-c019-4420-89aa-7468dadb0659-kube-api-access-l9j8p\" (UniqueName: \"kubernetes.io/projected/21d23e7c-c019-4420-89aa-7468dadb0659-kube-api-access-l9j8p\") pod \"21d23e7c-c019-4420-89aa-7468dadb0659\" (UID: \"21d23e7c-c019-4420-89aa-7468dadb0659\") " Oct 30 01:39:08.573113 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:39:08.572974 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21d23e7c-c019-4420-89aa-7468dadb0659-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d23e7c-c019-4420-89aa-7468dadb0659-plugins-config-volume\") pod \"21d23e7c-c019-4420-89aa-7468dadb0659\" (UID: \"21d23e7c-c019-4420-89aa-7468dadb0659\") " Oct 30 01:39:08.573449 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:39:08.573420 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d23e7c-c019-4420-89aa-7468dadb0659-plugins-config-volume" pod "21d23e7c-c019-4420-89aa-7468dadb0659" (UID: "21d23e7c-c019-4420-89aa-7468dadb0659"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:39:08.575219 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:39:08.575195 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d23e7c-c019-4420-89aa-7468dadb0659-kube-api-access-l9j8p" pod "21d23e7c-c019-4420-89aa-7468dadb0659" (UID: "21d23e7c-c019-4420-89aa-7468dadb0659"). InnerVolumeSpecName "kube-api-access-l9j8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:39:08.674042 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:39:08.674005 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d23e7c-c019-4420-89aa-7468dadb0659-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:39:08.674042 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:39:08.674041 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9j8p\" (UniqueName: \"kubernetes.io/projected/21d23e7c-c019-4420-89aa-7468dadb0659-kube-api-access-l9j8p\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:39:09.191070 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:39:09.191033 5300 generic.go:356] "Generic (PLEG): container finished" podID="21d23e7c-c019-4420-89aa-7468dadb0659" containerID="b07ebdc003d798770a14a0f4b10fc2ae5315d69522937e3e91f019314cdce7ed" exitCode=0 Oct 30 01:39:09.191269 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:39:09.191092 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b" Oct 30 01:39:09.191269 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:39:09.191094 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b" event={"ID":"21d23e7c-c019-4420-89aa-7468dadb0659","Type":"ContainerDied","Data":"b07ebdc003d798770a14a0f4b10fc2ae5315d69522937e3e91f019314cdce7ed"} Oct 30 01:39:09.191269 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:39:09.191228 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b" event={"ID":"21d23e7c-c019-4420-89aa-7468dadb0659","Type":"ContainerDied","Data":"de740349eb58925596e54e75a0fbb2e7bce220d1091c03f0b2ee1f5e91900396"} Oct 30 01:39:09.191269 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:39:09.191259 5300 scope.go:122] "RemoveContainer" containerID="b07ebdc003d798770a14a0f4b10fc2ae5315d69522937e3e91f019314cdce7ed" Oct 30 01:39:09.200274 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:39:09.200254 5300 scope.go:122] "RemoveContainer" containerID="b07ebdc003d798770a14a0f4b10fc2ae5315d69522937e3e91f019314cdce7ed" Oct 30 01:39:09.200558 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:39:09.200532 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b07ebdc003d798770a14a0f4b10fc2ae5315d69522937e3e91f019314cdce7ed\": container with ID starting with b07ebdc003d798770a14a0f4b10fc2ae5315d69522937e3e91f019314cdce7ed not found: ID does not exist" containerID="b07ebdc003d798770a14a0f4b10fc2ae5315d69522937e3e91f019314cdce7ed" Oct 30 01:39:09.200632 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:39:09.200564 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b07ebdc003d798770a14a0f4b10fc2ae5315d69522937e3e91f019314cdce7ed"} err="failed to get container status \"b07ebdc003d798770a14a0f4b10fc2ae5315d69522937e3e91f019314cdce7ed\": rpc error: code = NotFound desc = could not find container \"b07ebdc003d798770a14a0f4b10fc2ae5315d69522937e3e91f019314cdce7ed\": container with ID starting with b07ebdc003d798770a14a0f4b10fc2ae5315d69522937e3e91f019314cdce7ed not found: ID does not exist" Oct 30 01:39:09.248931 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:39:09.248910 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b"] Oct 30 01:39:09.262932 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:39:09.262908 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-37ba51-59fcb8d54zlv2b"] Oct 30 01:39:09.819959 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:39:09.819923 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21d23e7c-c019-4420-89aa-7468dadb0659" path="/var/lib/kubelet/pods/21d23e7c-c019-4420-89aa-7468dadb0659/volumes" Oct 30 01:39:13.823067 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:39:13.823032 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:15.389901 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:39:15.389863 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:20.898383 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:39:20.898343 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:21.591918 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:39:21.591882 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:39:36.577523 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:39:36.577488 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:51.592502 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:39:51.592463 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:40:21.592197 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:40:21.592158 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:40:36.898032 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:40:36.897948 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:51.591787 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:40:51.591737 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:41:04.870178 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:04.870115 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:05.274629 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.274592 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms"] Oct 30 01:41:05.275184 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.275157 5300 cpu_manager.go:414] "containerMap: removing container" podUID="ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f" containerName="istio-proxy" Oct 30 01:41:05.275184 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.275183 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f" containerName="istio-proxy" Oct 30 01:41:05.275383 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.275218 5300 cpu_manager.go:414] "containerMap: removing container" podUID="ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f" containerName="istio-proxy" Oct 30 01:41:05.275383 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.275226 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f" containerName="istio-proxy" Oct 30 01:41:05.275383 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.275248 5300 cpu_manager.go:414] "containerMap: removing container" podUID="21d23e7c-c019-4420-89aa-7468dadb0659" containerName="payload-processing" Oct 30 01:41:05.275383 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.275256 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21d23e7c-c019-4420-89aa-7468dadb0659" containerName="payload-processing" Oct 30 01:41:05.275896 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.275392 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f" containerName="istio-proxy" Oct 30 01:41:05.275896 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.275409 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="21d23e7c-c019-4420-89aa-7468dadb0659" containerName="payload-processing" Oct 30 01:41:05.278749 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.278729 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.298525 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.298505 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-b63f8a-openshift-default-dockercfg-22zfz\"" Oct 30 01:41:05.343150 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.343124 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms"] Oct 30 01:41:05.404258 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.404221 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-credential-socket\") pod \"e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.404389 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.404266 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-token\") pod \"e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.404389 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.404293 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzpsl\" (UniqueName: \"kubernetes.io/projected/9194627b-c8ee-47a2-b72b-ac4d17cbedee-kube-api-access-bzpsl\") pod \"e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.404389 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.404354 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-data\") pod \"e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.404573 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.404452 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-workload-socket\") pod \"e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.404573 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.404508 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-workload-certs\") pod \"e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.404573 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.404542 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-envoy\") pod \"e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.404712 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.404637 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istiod-ca-cert\") pod \"e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.404712 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.404668 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-podinfo\") pod \"e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.412279 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.412250 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:05.506397 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.506358 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-workload-socket\") pod \"e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.506582 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.506444 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-workload-certs\") pod \"e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.506582 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.506470 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-envoy\") pod \"e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.506708 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.506596 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istiod-ca-cert\") pod \"e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.506708 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.506661 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-podinfo\") pod \"e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.506860 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.506754 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-workload-socket\") pod \"e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.506860 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.506823 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-workload-certs\") pod \"e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.506860 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.506837 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-credential-socket\") pod \"e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.507158 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.506882 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-token\") pod \"e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.507158 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.506907 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzpsl\" (UniqueName: \"kubernetes.io/projected/9194627b-c8ee-47a2-b72b-ac4d17cbedee-kube-api-access-bzpsl\") pod \"e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.507158 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.506945 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-data\") pod \"e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.507279 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.507183 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-data\") pod \"e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.507279 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.507261 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istiod-ca-cert\") pod \"e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.507621 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.507598 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-credential-socket\") pod \"e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.509032 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.509005 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-envoy\") pod \"e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.509286 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.509268 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-podinfo\") pod \"e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.520491 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.520415 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:05.527724 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.527670 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-token\") pod \"e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.528218 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.528196 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzpsl\" (UniqueName: \"kubernetes.io/projected/9194627b-c8ee-47a2-b72b-ac4d17cbedee-kube-api-access-bzpsl\") pod \"e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.591405 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.591384 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:05.900049 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.900028 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms"] Oct 30 01:41:05.901508 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:41:05.901481 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9194627b_c8ee_47a2_b72b_ac4d17cbedee.slice/crio-239fb7d993d67925e4011c06c19d2c9044bed7351b0c94b2b48b99788970deef WatchSource:0}: Error finding container 239fb7d993d67925e4011c06c19d2c9044bed7351b0c94b2b48b99788970deef: Status 404 returned error can't find the container with id 239fb7d993d67925e4011c06c19d2c9044bed7351b0c94b2b48b99788970deef Oct 30 01:41:05.903673 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.903633 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:41:05.903746 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.903710 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:41:05.903810 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:05.903753 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:41:06.744949 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:06.744916 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" event={"ID":"9194627b-c8ee-47a2-b72b-ac4d17cbedee","Type":"ContainerStarted","Data":"fbfd016af2bacda92cd36e1629c7b367a9a6b69723d4203e96ceb2ed45793918"} Oct 30 01:41:06.745117 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:06.744990 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" event={"ID":"9194627b-c8ee-47a2-b72b-ac4d17cbedee","Type":"ContainerStarted","Data":"239fb7d993d67925e4011c06c19d2c9044bed7351b0c94b2b48b99788970deef"} Oct 30 01:41:06.820144 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:06.820101 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" podStartSLOduration=1.82008942 podStartE2EDuration="1.82008942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:41:06.805284115 +0000 UTC m=+3467.121152443" watchObservedRunningTime="2026-10-30 01:41:06.82008942 +0000 UTC m=+3467.135957741" Oct 30 01:41:07.495365 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:07.495332 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:07.592391 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:07.592364 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:07.597409 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:07.597385 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:07.750351 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:07.750286 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:07.751131 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:07.751114 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:09.144566 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:09.144535 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:09.290487 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:09.290451 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:10.815485 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:10.815439 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:10.912016 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:10.911985 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:11.012002 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:11.011972 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:11.155227 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:11.155144 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd"] Oct 30 01:41:11.155832 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:11.155813 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba721bdc-fdb5-4a6a-ac43-e45f088b7c8f" containerName="istio-proxy" Oct 30 01:41:11.159441 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:11.159419 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd" Oct 30 01:41:11.176899 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:11.176862 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd"] Oct 30 01:41:11.188855 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:11.188820 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-b63f8a\"" Oct 30 01:41:11.188855 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:11.188829 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-640f0b93-dockercfg-4rhqd\"" Oct 30 01:41:11.260169 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:11.260132 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd\" (UID: \"7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd" Oct 30 01:41:11.260398 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:11.260184 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bvq8\" (UniqueName: \"kubernetes.io/projected/7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09-kube-api-access-5bvq8\") pod \"payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd\" (UID: \"7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd" Oct 30 01:41:11.361727 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:11.361688 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd\" (UID: \"7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd" Oct 30 01:41:11.361970 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:11.361751 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bvq8\" (UniqueName: \"kubernetes.io/projected/7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09-kube-api-access-5bvq8\") pod \"payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd\" (UID: \"7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd" Oct 30 01:41:11.362456 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:11.362434 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd\" (UID: \"7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd" Oct 30 01:41:11.383634 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:11.383598 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bvq8\" (UniqueName: \"kubernetes.io/projected/7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09-kube-api-access-5bvq8\") pod \"payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd\" (UID: \"7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd" Oct 30 01:41:11.475739 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:11.475711 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd" Oct 30 01:41:11.769598 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:11.769571 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd"] Oct 30 01:41:11.771981 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:41:11.771954 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f1ed9f1_ad2c_420c_a6e1_3bd58796bc09.slice/crio-7c588cad16db238f4f97b53fdbc247e6ac724e8ecb91a523706de06b03fbbc09 WatchSource:0}: Error finding container 7c588cad16db238f4f97b53fdbc247e6ac724e8ecb91a523706de06b03fbbc09: Status 404 returned error can't find the container with id 7c588cad16db238f4f97b53fdbc247e6ac724e8ecb91a523706de06b03fbbc09 Oct 30 01:41:12.705874 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:12.705842 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:12.780170 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:12.780127 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd" event={"ID":"7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09","Type":"ContainerStarted","Data":"7b478a34de696fe6d42b5d1d7136821553db2e61c2ed138474f07dd966bdf573"} Oct 30 01:41:12.780364 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:12.780214 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd" event={"ID":"7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09","Type":"ContainerStarted","Data":"7c588cad16db238f4f97b53fdbc247e6ac724e8ecb91a523706de06b03fbbc09"} Oct 30 01:41:12.780364 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:12.780336 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd" Oct 30 01:41:12.823211 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:12.823169 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd" podStartSLOduration=1.823158268 podStartE2EDuration="1.823158268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:41:12.822219756 +0000 UTC m=+3473.138088080" watchObservedRunningTime="2026-10-30 01:41:12.823158268 +0000 UTC m=+3473.139026588" Oct 30 01:41:14.504381 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:14.504348 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:16.137479 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:16.137424 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:17.843334 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:17.843279 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:19.363436 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:19.363394 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:21.056241 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:21.056219 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:21.592926 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:41:21.592883 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:41:22.765052 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:22.765012 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:23.788430 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:23.788402 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd" Oct 30 01:41:23.791866 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:23.791846 5300 patch_prober.go:29] interesting pod/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:41:23.791948 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:23.791886 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" podUID="9194627b-c8ee-47a2-b72b-ac4d17cbedee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:41:23.966991 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:23.966938 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:24.317611 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:24.317550 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:24.558798 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:24.558757 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:24.687348 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:24.687273 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:24.830179 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:24.830137 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:24.834959 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:24.834935 5300 generic.go:356] "Generic (PLEG): container finished" podID="9194627b-c8ee-47a2-b72b-ac4d17cbedee" containerID="fbfd016af2bacda92cd36e1629c7b367a9a6b69723d4203e96ceb2ed45793918" exitCode=0 Oct 30 01:41:24.835097 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:24.835002 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" event={"ID":"9194627b-c8ee-47a2-b72b-ac4d17cbedee","Type":"ContainerDied","Data":"fbfd016af2bacda92cd36e1629c7b367a9a6b69723d4203e96ceb2ed45793918"} Oct 30 01:41:24.835357 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:24.835344 5300 scope.go:122] "RemoveContainer" containerID="fbfd016af2bacda92cd36e1629c7b367a9a6b69723d4203e96ceb2ed45793918" Oct 30 01:41:24.836465 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:24.836436 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:41:24.836521 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:24.836506 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:41:24.836559 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:24.836538 5300 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 01:41:25.846497 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:25.846465 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" event={"ID":"9194627b-c8ee-47a2-b72b-ac4d17cbedee","Type":"ContainerStarted","Data":"f65aca2212a9145c6051cc96ede9fef9828cbc0b10355781aee98ef1a5792698"} Oct 30 01:41:26.242947 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:26.242907 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:26.591887 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:26.591798 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:26.592060 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:26.592012 5300 patch_prober.go:29] interesting pod/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 01:41:26.592121 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:26.592060 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" podUID="9194627b-c8ee-47a2-b72b-ac4d17cbedee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 01:41:27.199791 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:27.199748 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:27.592019 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:27.591991 5300 patch_prober.go:29] interesting pod/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 01:41:27.592191 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:27.592039 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" podUID="9194627b-c8ee-47a2-b72b-ac4d17cbedee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 01:41:27.980923 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:27.980886 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:28.592702 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:28.592665 5300 patch_prober.go:29] interesting pod/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 01:41:28.593200 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:28.592723 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" podUID="9194627b-c8ee-47a2-b72b-ac4d17cbedee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 01:41:29.592173 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:29.592135 5300 patch_prober.go:29] interesting pod/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 01:41:29.592372 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:29.592218 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" podUID="9194627b-c8ee-47a2-b72b-ac4d17cbedee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 01:41:29.671824 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:29.671785 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:30.592397 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:30.592363 5300 patch_prober.go:29] interesting pod/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 01:41:30.592583 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:30.592431 5300 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" podUID="9194627b-c8ee-47a2-b72b-ac4d17cbedee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 01:41:31.539996 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:31.539960 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:31.595861 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:31.595827 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:31.596092 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:31.596073 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:31.596690 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:31.596671 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:41:33.116399 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:33.116360 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:33.801027 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:33.800993 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:34.894699 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:34.894663 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:36.501661 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:36.501628 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:36.916594 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:36.916508 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:37.223695 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:37.223659 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:38.095016 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:38.094983 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:40.005417 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:40.005384 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:41.618807 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:41.618748 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:43.305532 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:43.305496 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:44.994780 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:44.994170 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:46.659989 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:46.659954 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:48.234951 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:48.234909 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:49.749798 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:49.749751 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:51.397790 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:51.397749 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:51.591318 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:41:51.591283 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:41:53.030643 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:53.030613 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:54.777946 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:54.777917 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:56.417208 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:56.417180 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:56.517716 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:56.517684 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:58.083149 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:58.083114 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:59.680658 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:41:59.680624 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:01.315393 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:01.315312 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:01.407397 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:01.407369 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:03.002425 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:03.002393 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:04.533014 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:04.532972 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:06.122397 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:06.122364 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:06.980978 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:06.980939 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd"] Oct 30 01:42:07.112288 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:07.112202 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:07.217241 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:07.217206 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:07.527487 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:07.527456 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:09.716019 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:09.715986 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:09.823215 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:09.823178 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:10.331176 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:10.331141 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:11.981973 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:11.981920 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd" podUID="7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09" containerName="payload-processing" containerID="cri-o://7b478a34de696fe6d42b5d1d7136821553db2e61c2ed138474f07dd966bdf573" gracePeriod=25 Oct 30 01:42:13.786301 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:13.786268 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd" podUID="7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:42:16.230069 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:16.230009 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:16.323788 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:16.323727 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:16.560368 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:16.560285 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms"] Oct 30 01:42:16.560845 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:16.560810 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" podUID="9194627b-c8ee-47a2-b72b-ac4d17cbedee" containerName="istio-proxy" containerID="cri-o://f65aca2212a9145c6051cc96ede9fef9828cbc0b10355781aee98ef1a5792698" gracePeriod=30 Oct 30 01:42:16.591780 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:16.591733 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:16.599283 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:16.599251 5300 patch_prober.go:29] interesting pod/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:42:16.599425 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:16.599301 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" podUID="9194627b-c8ee-47a2-b72b-ac4d17cbedee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:42:20.351550 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:20.351519 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:21.594356 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:42:21.594326 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:42:21.805942 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.805914 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:42:21.871066 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.870983 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9194627b-c8ee-47a2-b72b-ac4d17cbedee-kube-api-access-bzpsl\" (UniqueName: \"kubernetes.io/projected/9194627b-c8ee-47a2-b72b-ac4d17cbedee-kube-api-access-bzpsl\") pod \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " Oct 30 01:42:21.871066 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.871032 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-data\") pod \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " Oct 30 01:42:21.871320 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.871071 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-podinfo\") pod \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " Oct 30 01:42:21.871320 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.871156 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istiod-ca-cert\") pod \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " Oct 30 01:42:21.871320 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.871211 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-token\" (UniqueName: \"kubernetes.io/projected/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-token\") pod \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " Oct 30 01:42:21.871320 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.871238 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-workload-certs\") pod \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " Oct 30 01:42:21.871320 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.871284 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-workload-socket\") pod \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " Oct 30 01:42:21.872177 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.871340 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-credential-socket\") pod \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " Oct 30 01:42:21.872177 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.871351 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-data" pod "9194627b-c8ee-47a2-b72b-ac4d17cbedee" (UID: "9194627b-c8ee-47a2-b72b-ac4d17cbedee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:21.872177 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.871367 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-envoy\") pod \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\" (UID: \"9194627b-c8ee-47a2-b72b-ac4d17cbedee\") " Oct 30 01:42:21.872177 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.871539 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-workload-socket" pod "9194627b-c8ee-47a2-b72b-ac4d17cbedee" (UID: "9194627b-c8ee-47a2-b72b-ac4d17cbedee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:21.872177 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.871746 5300 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-workload-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:42:21.872177 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.871792 5300 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-data\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:42:21.872177 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.872012 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-workload-certs" pod "9194627b-c8ee-47a2-b72b-ac4d17cbedee" (UID: "9194627b-c8ee-47a2-b72b-ac4d17cbedee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:21.872177 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.872026 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-credential-socket" pod "9194627b-c8ee-47a2-b72b-ac4d17cbedee" (UID: "9194627b-c8ee-47a2-b72b-ac4d17cbedee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:21.872665 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.872320 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istiod-ca-cert" pod "9194627b-c8ee-47a2-b72b-ac4d17cbedee" (UID: "9194627b-c8ee-47a2-b72b-ac4d17cbedee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:42:21.874501 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.874479 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9194627b-c8ee-47a2-b72b-ac4d17cbedee-kube-api-access-bzpsl" pod "9194627b-c8ee-47a2-b72b-ac4d17cbedee" (UID: "9194627b-c8ee-47a2-b72b-ac4d17cbedee"). InnerVolumeSpecName "kube-api-access-bzpsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:42:21.874637 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.874614 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-token" pod "9194627b-c8ee-47a2-b72b-ac4d17cbedee" (UID: "9194627b-c8ee-47a2-b72b-ac4d17cbedee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:42:21.874698 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.874614 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-envoy" pod "9194627b-c8ee-47a2-b72b-ac4d17cbedee" (UID: "9194627b-c8ee-47a2-b72b-ac4d17cbedee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:21.875265 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.875245 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-podinfo" pod "9194627b-c8ee-47a2-b72b-ac4d17cbedee" (UID: "9194627b-c8ee-47a2-b72b-ac4d17cbedee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:42:21.972566 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.972539 5300 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-podinfo\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:42:21.972566 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.972563 5300 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istiod-ca-cert\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:42:21.972566 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.972572 5300 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-token\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:42:21.972795 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.972579 5300 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-workload-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:42:21.972795 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.972588 5300 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-credential-socket\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:42:21.972795 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.972596 5300 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9194627b-c8ee-47a2-b72b-ac4d17cbedee-istio-envoy\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:42:21.972795 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:21.972604 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzpsl\" (UniqueName: \"kubernetes.io/projected/9194627b-c8ee-47a2-b72b-ac4d17cbedee-kube-api-access-bzpsl\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:42:22.100480 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:22.100441 5300 generic.go:356] "Generic (PLEG): container finished" podID="9194627b-c8ee-47a2-b72b-ac4d17cbedee" containerID="f65aca2212a9145c6051cc96ede9fef9828cbc0b10355781aee98ef1a5792698" exitCode=0 Oct 30 01:42:22.100659 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:22.100513 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" Oct 30 01:42:22.100659 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:22.100527 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" event={"ID":"9194627b-c8ee-47a2-b72b-ac4d17cbedee","Type":"ContainerDied","Data":"f65aca2212a9145c6051cc96ede9fef9828cbc0b10355781aee98ef1a5792698"} Oct 30 01:42:22.100659 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:22.100605 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms" event={"ID":"9194627b-c8ee-47a2-b72b-ac4d17cbedee","Type":"ContainerDied","Data":"239fb7d993d67925e4011c06c19d2c9044bed7351b0c94b2b48b99788970deef"} Oct 30 01:42:22.100659 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:22.100627 5300 scope.go:122] "RemoveContainer" containerID="f65aca2212a9145c6051cc96ede9fef9828cbc0b10355781aee98ef1a5792698" Oct 30 01:42:22.111990 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:22.111946 5300 scope.go:122] "RemoveContainer" containerID="fbfd016af2bacda92cd36e1629c7b367a9a6b69723d4203e96ceb2ed45793918" Oct 30 01:42:22.120209 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:22.120186 5300 scope.go:122] "RemoveContainer" containerID="f65aca2212a9145c6051cc96ede9fef9828cbc0b10355781aee98ef1a5792698" Oct 30 01:42:22.120465 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:42:22.120448 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f65aca2212a9145c6051cc96ede9fef9828cbc0b10355781aee98ef1a5792698\": container with ID starting with f65aca2212a9145c6051cc96ede9fef9828cbc0b10355781aee98ef1a5792698 not found: ID does not exist" containerID="f65aca2212a9145c6051cc96ede9fef9828cbc0b10355781aee98ef1a5792698" Oct 30 01:42:22.120527 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:22.120475 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f65aca2212a9145c6051cc96ede9fef9828cbc0b10355781aee98ef1a5792698"} err="failed to get container status \"f65aca2212a9145c6051cc96ede9fef9828cbc0b10355781aee98ef1a5792698\": rpc error: code = NotFound desc = could not find container \"f65aca2212a9145c6051cc96ede9fef9828cbc0b10355781aee98ef1a5792698\": container with ID starting with f65aca2212a9145c6051cc96ede9fef9828cbc0b10355781aee98ef1a5792698 not found: ID does not exist" Oct 30 01:42:22.120527 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:22.120490 5300 scope.go:122] "RemoveContainer" containerID="fbfd016af2bacda92cd36e1629c7b367a9a6b69723d4203e96ceb2ed45793918" Oct 30 01:42:22.120741 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:42:22.120724 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbfd016af2bacda92cd36e1629c7b367a9a6b69723d4203e96ceb2ed45793918\": container with ID starting with fbfd016af2bacda92cd36e1629c7b367a9a6b69723d4203e96ceb2ed45793918 not found: ID does not exist" containerID="fbfd016af2bacda92cd36e1629c7b367a9a6b69723d4203e96ceb2ed45793918" Oct 30 01:42:22.120822 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:22.120744 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbfd016af2bacda92cd36e1629c7b367a9a6b69723d4203e96ceb2ed45793918"} err="failed to get container status \"fbfd016af2bacda92cd36e1629c7b367a9a6b69723d4203e96ceb2ed45793918\": rpc error: code = NotFound desc = could not find container \"fbfd016af2bacda92cd36e1629c7b367a9a6b69723d4203e96ceb2ed45793918\": container with ID starting with fbfd016af2bacda92cd36e1629c7b367a9a6b69723d4203e96ceb2ed45793918 not found: ID does not exist" Oct 30 01:42:22.157056 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:22.156993 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms"] Oct 30 01:42:22.171168 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:22.171149 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b63f8a-openshift-default-6876df6cb9nhtms"] Oct 30 01:42:23.786803 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:23.786742 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd" podUID="7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:42:23.822204 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:23.822169 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9194627b-c8ee-47a2-b72b-ac4d17cbedee" path="/var/lib/kubelet/pods/9194627b-c8ee-47a2-b72b-ac4d17cbedee/volumes" Oct 30 01:42:32.149207 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:32.149165 5300 generic.go:356] "Generic (PLEG): container finished" podID="7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09" containerID="7b478a34de696fe6d42b5d1d7136821553db2e61c2ed138474f07dd966bdf573" exitCode=0 Oct 30 01:42:32.149587 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:32.149230 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd" event={"ID":"7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09","Type":"ContainerDied","Data":"7b478a34de696fe6d42b5d1d7136821553db2e61c2ed138474f07dd966bdf573"} Oct 30 01:42:32.232365 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:32.232341 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd" Oct 30 01:42:32.375637 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:32.375552 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09-plugins-config-volume\") pod \"7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09\" (UID: \"7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09\") " Oct 30 01:42:32.375820 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:32.375685 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09-kube-api-access-5bvq8\" (UniqueName: \"kubernetes.io/projected/7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09-kube-api-access-5bvq8\") pod \"7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09\" (UID: \"7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09\") " Oct 30 01:42:32.375974 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:32.375953 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09-plugins-config-volume" pod "7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09" (UID: "7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:42:32.377960 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:32.377933 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09-kube-api-access-5bvq8" pod "7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09" (UID: "7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09"). InnerVolumeSpecName "kube-api-access-5bvq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:42:32.477110 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:32.477071 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bvq8\" (UniqueName: \"kubernetes.io/projected/7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09-kube-api-access-5bvq8\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:42:32.477110 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:32.477103 5300 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09-plugins-config-volume\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:42:33.155493 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:33.155454 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd" event={"ID":"7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09","Type":"ContainerDied","Data":"7c588cad16db238f4f97b53fdbc247e6ac724e8ecb91a523706de06b03fbbc09"} Oct 30 01:42:33.155493 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:33.155476 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd" Oct 30 01:42:33.156023 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:33.155521 5300 scope.go:122] "RemoveContainer" containerID="7b478a34de696fe6d42b5d1d7136821553db2e61c2ed138474f07dd966bdf573" Oct 30 01:42:33.211111 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:33.211080 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd"] Oct 30 01:42:33.224980 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:33.224957 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b63f8a-7b566c5bdrdpxd"] Oct 30 01:42:33.818789 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:33.818733 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09" path="/var/lib/kubelet/pods/7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09/volumes" Oct 30 01:42:37.027413 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:37.027358 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:37.124576 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:37.124546 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:37.698347 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:37.698318 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:39.197840 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.197806 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd"] Oct 30 01:42:39.198238 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.198172 5300 cpu_manager.go:414] "containerMap: removing container" podUID="9194627b-c8ee-47a2-b72b-ac4d17cbedee" containerName="istio-proxy" Oct 30 01:42:39.198238 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.198183 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9194627b-c8ee-47a2-b72b-ac4d17cbedee" containerName="istio-proxy" Oct 30 01:42:39.198238 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.198207 5300 cpu_manager.go:414] "containerMap: removing container" podUID="7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09" containerName="payload-processing" Oct 30 01:42:39.198238 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.198212 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09" containerName="payload-processing" Oct 30 01:42:39.198371 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.198270 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="9194627b-c8ee-47a2-b72b-ac4d17cbedee" containerName="istio-proxy" Oct 30 01:42:39.198371 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.198281 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f1ed9f1-ad2c-420c-a6e1-3bd58796bc09" containerName="payload-processing" Oct 30 01:42:39.198371 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.198289 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="9194627b-c8ee-47a2-b72b-ac4d17cbedee" containerName="istio-proxy" Oct 30 01:42:39.203067 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.203043 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" Oct 30 01:42:39.221319 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.221292 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd"] Oct 30 01:42:39.234445 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.234416 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-546f251f-kserve-self-signed-certs\"" Oct 30 01:42:39.258328 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.258279 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:39.343045 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.342998 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzbw6\" (UniqueName: \"kubernetes.io/projected/c63d18cc-c5a8-4758-8396-1c3823044238-kube-api-access-mzbw6\") pod \"e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd\" (UID: \"c63d18cc-c5a8-4758-8396-1c3823044238\") " pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" Oct 30 01:42:39.343045 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.343049 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-dshm\") pod \"e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd\" (UID: \"c63d18cc-c5a8-4758-8396-1c3823044238\") " pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" Oct 30 01:42:39.343307 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.343205 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-tmp-dir\") pod \"e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd\" (UID: \"c63d18cc-c5a8-4758-8396-1c3823044238\") " pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" Oct 30 01:42:39.343307 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.343240 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63d18cc-c5a8-4758-8396-1c3823044238-tls-certs\") pod \"e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd\" (UID: \"c63d18cc-c5a8-4758-8396-1c3823044238\") " pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" Oct 30 01:42:39.343307 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.343279 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-model-cache\") pod \"e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd\" (UID: \"c63d18cc-c5a8-4758-8396-1c3823044238\") " pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" Oct 30 01:42:39.343441 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.343342 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-home\") pod \"e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd\" (UID: \"c63d18cc-c5a8-4758-8396-1c3823044238\") " pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" Oct 30 01:42:39.444133 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.444094 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-tmp-dir\") pod \"e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd\" (UID: \"c63d18cc-c5a8-4758-8396-1c3823044238\") " pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" Oct 30 01:42:39.444327 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.444141 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63d18cc-c5a8-4758-8396-1c3823044238-tls-certs\") pod \"e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd\" (UID: \"c63d18cc-c5a8-4758-8396-1c3823044238\") " pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" Oct 30 01:42:39.444327 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.444180 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-model-cache\") pod \"e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd\" (UID: \"c63d18cc-c5a8-4758-8396-1c3823044238\") " pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" Oct 30 01:42:39.444327 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.444234 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-home\") pod \"e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd\" (UID: \"c63d18cc-c5a8-4758-8396-1c3823044238\") " pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" Oct 30 01:42:39.444327 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.444291 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzbw6\" (UniqueName: \"kubernetes.io/projected/c63d18cc-c5a8-4758-8396-1c3823044238-kube-api-access-mzbw6\") pod \"e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd\" (UID: \"c63d18cc-c5a8-4758-8396-1c3823044238\") " pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" Oct 30 01:42:39.444327 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.444320 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-dshm\") pod \"e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd\" (UID: \"c63d18cc-c5a8-4758-8396-1c3823044238\") " pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" Oct 30 01:42:39.444686 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.444600 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-tmp-dir\") pod \"e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd\" (UID: \"c63d18cc-c5a8-4758-8396-1c3823044238\") " pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" Oct 30 01:42:39.444686 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.444646 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-model-cache\") pod \"e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd\" (UID: \"c63d18cc-c5a8-4758-8396-1c3823044238\") " pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" Oct 30 01:42:39.444844 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.444824 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-home\") pod \"e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd\" (UID: \"c63d18cc-c5a8-4758-8396-1c3823044238\") " pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" Oct 30 01:42:39.446619 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.446595 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-dshm\") pod \"e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd\" (UID: \"c63d18cc-c5a8-4758-8396-1c3823044238\") " pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" Oct 30 01:42:39.446800 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.446776 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63d18cc-c5a8-4758-8396-1c3823044238-tls-certs\") pod \"e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd\" (UID: \"c63d18cc-c5a8-4758-8396-1c3823044238\") " pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" Oct 30 01:42:39.474935 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.474906 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzbw6\" (UniqueName: \"kubernetes.io/projected/c63d18cc-c5a8-4758-8396-1c3823044238-kube-api-access-mzbw6\") pod \"e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd\" (UID: \"c63d18cc-c5a8-4758-8396-1c3823044238\") " pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" Oct 30 01:42:39.516283 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.516247 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" Oct 30 01:42:39.812181 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:39.812155 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd"] Oct 30 01:42:39.814959 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:42:39.814925 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc63d18cc_c5a8_4758_8396_1c3823044238.slice/crio-67d5a05f632e69f0863cee283bd64d08be1946aafb5ec1d2865096da675dea2a WatchSource:0}: Error finding container 67d5a05f632e69f0863cee283bd64d08be1946aafb5ec1d2865096da675dea2a: Status 404 returned error can't find the container with id 67d5a05f632e69f0863cee283bd64d08be1946aafb5ec1d2865096da675dea2a Oct 30 01:42:40.188311 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:40.188214 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" event={"ID":"c63d18cc-c5a8-4758-8396-1c3823044238","Type":"ContainerStarted","Data":"5a09ce73d14cf4e874191db526a5943509cd16eaabeba6fe0c44d758b84b607e"} Oct 30 01:42:40.188311 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:40.188257 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" event={"ID":"c63d18cc-c5a8-4758-8396-1c3823044238","Type":"ContainerStarted","Data":"67d5a05f632e69f0863cee283bd64d08be1946aafb5ec1d2865096da675dea2a"} Oct 30 01:42:40.235828 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:40.235710 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" podStartSLOduration=1.235695108 podStartE2EDuration="1.235695108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:42:40.234943988 +0000 UTC m=+3560.550812313" watchObservedRunningTime="2026-10-30 01:42:40.235695108 +0000 UTC m=+3560.551563443" Oct 30 01:42:40.517087 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:40.517053 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" Oct 30 01:42:44.207588 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:44.207555 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:44.941785 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:44.941700 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:45.037783 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:45.037120 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:47.865531 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:47.865494 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:47.965387 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:47.965360 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:49.517082 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:49.517048 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" Oct 30 01:42:49.529478 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:49.529451 5300 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" Oct 30 01:42:50.249574 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:50.249549 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" Oct 30 01:42:51.592574 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:42:51.592538 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:42:58.138918 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:58.138853 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dzvm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:42:58.139590 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:42:58.139557 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" podUID="5f16e1d3-330a-4e81-b059-107bf40670b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:43:07.487123 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:07.487086 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:08.292245 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:08.292211 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd"] Oct 30 01:43:08.330565 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:08.330499 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:13.139784 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:13.138887 5300 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dzvm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:43:13.139784 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:13.139269 5300 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dzvm2" podUID="5f16e1d3-330a-4e81-b059-107bf40670b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:43:21.593116 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:43:21.593070 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:43:23.320600 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:23.320557 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" podUID="c63d18cc-c5a8-4758-8396-1c3823044238" containerName="main" containerID="cri-o://5a09ce73d14cf4e874191db526a5943509cd16eaabeba6fe0c44d758b84b607e" gracePeriod=45 Oct 30 01:43:23.475426 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:23.475391 5300 generic.go:356] "Generic (PLEG): container finished" podID="c63d18cc-c5a8-4758-8396-1c3823044238" containerID="5a09ce73d14cf4e874191db526a5943509cd16eaabeba6fe0c44d758b84b607e" exitCode=0 Oct 30 01:43:23.475591 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:23.475466 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" event={"ID":"c63d18cc-c5a8-4758-8396-1c3823044238","Type":"ContainerDied","Data":"5a09ce73d14cf4e874191db526a5943509cd16eaabeba6fe0c44d758b84b607e"} Oct 30 01:43:23.573858 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:23.573802 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" Oct 30 01:43:23.658988 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:23.658959 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-home\" (UniqueName: \"kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-home\") pod \"c63d18cc-c5a8-4758-8396-1c3823044238\" (UID: \"c63d18cc-c5a8-4758-8396-1c3823044238\") " Oct 30 01:43:23.659173 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:23.659013 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c63d18cc-c5a8-4758-8396-1c3823044238-kube-api-access-mzbw6\" (UniqueName: \"kubernetes.io/projected/c63d18cc-c5a8-4758-8396-1c3823044238-kube-api-access-mzbw6\") pod \"c63d18cc-c5a8-4758-8396-1c3823044238\" (UID: \"c63d18cc-c5a8-4758-8396-1c3823044238\") " Oct 30 01:43:23.659173 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:23.659070 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-tmp-dir\") pod \"c63d18cc-c5a8-4758-8396-1c3823044238\" (UID: \"c63d18cc-c5a8-4758-8396-1c3823044238\") " Oct 30 01:43:23.659173 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:23.659100 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c63d18cc-c5a8-4758-8396-1c3823044238-tls-certs\" (UniqueName: \"kubernetes.io/secret/c63d18cc-c5a8-4758-8396-1c3823044238-tls-certs\") pod \"c63d18cc-c5a8-4758-8396-1c3823044238\" (UID: \"c63d18cc-c5a8-4758-8396-1c3823044238\") " Oct 30 01:43:23.659173 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:23.659135 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-model-cache\") pod \"c63d18cc-c5a8-4758-8396-1c3823044238\" (UID: \"c63d18cc-c5a8-4758-8396-1c3823044238\") " Oct 30 01:43:23.659387 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:23.659202 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-dshm\") pod \"c63d18cc-c5a8-4758-8396-1c3823044238\" (UID: \"c63d18cc-c5a8-4758-8396-1c3823044238\") " Oct 30 01:43:23.659387 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:23.659218 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-home" pod "c63d18cc-c5a8-4758-8396-1c3823044238" (UID: "c63d18cc-c5a8-4758-8396-1c3823044238"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:23.659387 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:23.659354 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-tmp-dir" pod "c63d18cc-c5a8-4758-8396-1c3823044238" (UID: "c63d18cc-c5a8-4758-8396-1c3823044238"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:23.659520 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:23.659431 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-model-cache" pod "c63d18cc-c5a8-4758-8396-1c3823044238" (UID: "c63d18cc-c5a8-4758-8396-1c3823044238"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:23.659588 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:23.659569 5300 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-home\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:43:23.659644 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:23.659588 5300 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-tmp-dir\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:43:23.659644 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:23.659598 5300 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-model-cache\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:43:23.661400 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:23.661372 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-dshm" pod "c63d18cc-c5a8-4758-8396-1c3823044238" (UID: "c63d18cc-c5a8-4758-8396-1c3823044238"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:23.661400 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:23.661378 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63d18cc-c5a8-4758-8396-1c3823044238-kube-api-access-mzbw6" pod "c63d18cc-c5a8-4758-8396-1c3823044238" (UID: "c63d18cc-c5a8-4758-8396-1c3823044238"). InnerVolumeSpecName "kube-api-access-mzbw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:23.661400 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:23.661377 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c63d18cc-c5a8-4758-8396-1c3823044238-tls-certs" pod "c63d18cc-c5a8-4758-8396-1c3823044238" (UID: "c63d18cc-c5a8-4758-8396-1c3823044238"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:43:23.760987 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:23.760960 5300 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63d18cc-c5a8-4758-8396-1c3823044238-dshm\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:43:23.760987 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:23.760985 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzbw6\" (UniqueName: \"kubernetes.io/projected/c63d18cc-c5a8-4758-8396-1c3823044238-kube-api-access-mzbw6\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:43:23.760987 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:23.760997 5300 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63d18cc-c5a8-4758-8396-1c3823044238-tls-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:43:24.480845 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:24.480805 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" event={"ID":"c63d18cc-c5a8-4758-8396-1c3823044238","Type":"ContainerDied","Data":"67d5a05f632e69f0863cee283bd64d08be1946aafb5ec1d2865096da675dea2a"} Oct 30 01:43:24.480845 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:24.480845 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd" Oct 30 01:43:24.481337 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:24.480865 5300 scope.go:122] "RemoveContainer" containerID="5a09ce73d14cf4e874191db526a5943509cd16eaabeba6fe0c44d758b84b607e" Oct 30 01:43:24.535989 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:24.535964 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd"] Oct 30 01:43:24.549982 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:24.549955 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-546f251f-kserve-58d8856d8f-nhzkd"] Oct 30 01:43:25.819928 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:25.819894 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c63d18cc-c5a8-4758-8396-1c3823044238" path="/var/lib/kubelet/pods/c63d18cc-c5a8-4758-8396-1c3823044238/volumes" Oct 30 01:43:37.170502 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:37.170422 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:49.554184 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:49.554139 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-k46sj"] Oct 30 01:43:49.555830 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:49.555753 5300 cpu_manager.go:414] "containerMap: removing container" podUID="9194627b-c8ee-47a2-b72b-ac4d17cbedee" containerName="istio-proxy" Oct 30 01:43:49.556007 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:49.555992 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9194627b-c8ee-47a2-b72b-ac4d17cbedee" containerName="istio-proxy" Oct 30 01:43:49.556138 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:49.556127 5300 cpu_manager.go:414] "containerMap: removing container" podUID="c63d18cc-c5a8-4758-8396-1c3823044238" containerName="main" Oct 30 01:43:49.556221 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:49.556210 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63d18cc-c5a8-4758-8396-1c3823044238" containerName="main" Oct 30 01:43:49.556555 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:49.556537 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="c63d18cc-c5a8-4758-8396-1c3823044238" containerName="main" Oct 30 01:43:49.562144 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:49.562111 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-k46sj" Oct 30 01:43:49.635894 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:49.635865 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdf8e872-44f9-4c96-9ee4-ff00082048bd-maas-api-tls\") pod \"maas-api-595cc648c-k46sj\" (UID: \"bdf8e872-44f9-4c96-9ee4-ff00082048bd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-k46sj" Oct 30 01:43:49.636047 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:49.635906 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klqf9\" (UniqueName: \"kubernetes.io/projected/bdf8e872-44f9-4c96-9ee4-ff00082048bd-kube-api-access-klqf9\") pod \"maas-api-595cc648c-k46sj\" (UID: \"bdf8e872-44f9-4c96-9ee4-ff00082048bd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-k46sj" Oct 30 01:43:49.636047 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:49.635931 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdf8e872-44f9-4c96-9ee4-ff00082048bd-metrics-certs\") pod \"maas-api-595cc648c-k46sj\" (UID: \"bdf8e872-44f9-4c96-9ee4-ff00082048bd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-k46sj" Oct 30 01:43:49.636047 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:49.636033 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf8e872-44f9-4c96-9ee4-ff00082048bd-openshift-service-ca\") pod \"maas-api-595cc648c-k46sj\" (UID: \"bdf8e872-44f9-4c96-9ee4-ff00082048bd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-k46sj" Oct 30 01:43:49.654247 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:49.654222 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-k46sj"] Oct 30 01:43:49.737308 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:49.737272 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdf8e872-44f9-4c96-9ee4-ff00082048bd-metrics-certs\") pod \"maas-api-595cc648c-k46sj\" (UID: \"bdf8e872-44f9-4c96-9ee4-ff00082048bd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-k46sj" Oct 30 01:43:49.737478 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:49.737324 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf8e872-44f9-4c96-9ee4-ff00082048bd-openshift-service-ca\") pod \"maas-api-595cc648c-k46sj\" (UID: \"bdf8e872-44f9-4c96-9ee4-ff00082048bd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-k46sj" Oct 30 01:43:49.737478 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:49.737438 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdf8e872-44f9-4c96-9ee4-ff00082048bd-maas-api-tls\") pod \"maas-api-595cc648c-k46sj\" (UID: \"bdf8e872-44f9-4c96-9ee4-ff00082048bd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-k46sj" Oct 30 01:43:49.737478 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:49.737473 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klqf9\" (UniqueName: \"kubernetes.io/projected/bdf8e872-44f9-4c96-9ee4-ff00082048bd-kube-api-access-klqf9\") pod \"maas-api-595cc648c-k46sj\" (UID: \"bdf8e872-44f9-4c96-9ee4-ff00082048bd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-k46sj" Oct 30 01:43:49.738114 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:49.738090 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf8e872-44f9-4c96-9ee4-ff00082048bd-openshift-service-ca\") pod \"maas-api-595cc648c-k46sj\" (UID: \"bdf8e872-44f9-4c96-9ee4-ff00082048bd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-k46sj" Oct 30 01:43:49.739677 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:49.739651 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdf8e872-44f9-4c96-9ee4-ff00082048bd-metrics-certs\") pod \"maas-api-595cc648c-k46sj\" (UID: \"bdf8e872-44f9-4c96-9ee4-ff00082048bd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-k46sj" Oct 30 01:43:49.739826 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:49.739807 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdf8e872-44f9-4c96-9ee4-ff00082048bd-maas-api-tls\") pod \"maas-api-595cc648c-k46sj\" (UID: \"bdf8e872-44f9-4c96-9ee4-ff00082048bd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-k46sj" Oct 30 01:43:49.759396 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:49.759371 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klqf9\" (UniqueName: \"kubernetes.io/projected/bdf8e872-44f9-4c96-9ee4-ff00082048bd-kube-api-access-klqf9\") pod \"maas-api-595cc648c-k46sj\" (UID: \"bdf8e872-44f9-4c96-9ee4-ff00082048bd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-k46sj" Oct 30 01:43:49.875647 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:49.875571 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-k46sj" Oct 30 01:43:50.175853 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:50.175791 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-k46sj"] Oct 30 01:43:50.178139 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:43:50.178111 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf8e872_44f9_4c96_9ee4_ff00082048bd.slice/crio-487292e27de020c57dad3e564d2c5cde4ce2000c02945ec31ad9c1d0dc6c3f01 WatchSource:0}: Error finding container 487292e27de020c57dad3e564d2c5cde4ce2000c02945ec31ad9c1d0dc6c3f01: Status 404 returned error can't find the container with id 487292e27de020c57dad3e564d2c5cde4ce2000c02945ec31ad9c1d0dc6c3f01 Oct 30 01:43:50.179418 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:50.179402 5300 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:43:50.598009 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:50.597974 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-k46sj" event={"ID":"bdf8e872-44f9-4c96-9ee4-ff00082048bd","Type":"ContainerStarted","Data":"487292e27de020c57dad3e564d2c5cde4ce2000c02945ec31ad9c1d0dc6c3f01"} Oct 30 01:43:51.592491 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:43:51.592456 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:43:51.603807 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:51.603741 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-k46sj" event={"ID":"bdf8e872-44f9-4c96-9ee4-ff00082048bd","Type":"ContainerStarted","Data":"bf8dc3858749323a1ed12697c66d95e50e39b3364a0664a64e3aab89e1e499cd"} Oct 30 01:43:51.604134 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:51.603827 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-k46sj" Oct 30 01:43:51.670149 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:51.670107 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-k46sj" podStartSLOduration=2.244979278 podStartE2EDuration="2.670094241s" totalImagesPullingTime="425.114963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:43:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:43:51.668365074 +0000 UTC m=+3631.984233399" watchObservedRunningTime="2026-10-30 01:43:51.670094241 +0000 UTC m=+3631.985962564" Oct 30 01:43:57.613556 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:57.613525 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-k46sj" Oct 30 01:43:57.697330 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:57.697300 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln"] Oct 30 01:43:57.697509 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:57.697476 5300 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln" podUID="57c9985b-f3ab-4e84-8db9-41a954d358a6" containerName="maas-api" containerID="cri-o://e507abfe333c7bc21bbf1009b178bc6b037a51de612e94e8ddf38afcf456b8e7" gracePeriod=30 Oct 30 01:43:58.982498 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:58.982464 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln" Oct 30 01:43:59.016821 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:59.016792 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57c9985b-f3ab-4e84-8db9-41a954d358a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c9985b-f3ab-4e84-8db9-41a954d358a6-openshift-service-ca\") pod \"57c9985b-f3ab-4e84-8db9-41a954d358a6\" (UID: \"57c9985b-f3ab-4e84-8db9-41a954d358a6\") " Oct 30 01:43:59.017030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:59.016837 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57c9985b-f3ab-4e84-8db9-41a954d358a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c9985b-f3ab-4e84-8db9-41a954d358a6-metrics-certs\") pod \"57c9985b-f3ab-4e84-8db9-41a954d358a6\" (UID: \"57c9985b-f3ab-4e84-8db9-41a954d358a6\") " Oct 30 01:43:59.017030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:59.016858 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57c9985b-f3ab-4e84-8db9-41a954d358a6-kube-api-access-wg9rm\" (UniqueName: \"kubernetes.io/projected/57c9985b-f3ab-4e84-8db9-41a954d358a6-kube-api-access-wg9rm\") pod \"57c9985b-f3ab-4e84-8db9-41a954d358a6\" (UID: \"57c9985b-f3ab-4e84-8db9-41a954d358a6\") " Oct 30 01:43:59.017030 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:59.016888 5300 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57c9985b-f3ab-4e84-8db9-41a954d358a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c9985b-f3ab-4e84-8db9-41a954d358a6-maas-api-tls\") pod \"57c9985b-f3ab-4e84-8db9-41a954d358a6\" (UID: \"57c9985b-f3ab-4e84-8db9-41a954d358a6\") " Oct 30 01:43:59.017680 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:59.017637 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c9985b-f3ab-4e84-8db9-41a954d358a6-openshift-service-ca" pod "57c9985b-f3ab-4e84-8db9-41a954d358a6" (UID: "57c9985b-f3ab-4e84-8db9-41a954d358a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:43:59.019106 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:59.019076 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57c9985b-f3ab-4e84-8db9-41a954d358a6-metrics-certs" pod "57c9985b-f3ab-4e84-8db9-41a954d358a6" (UID: "57c9985b-f3ab-4e84-8db9-41a954d358a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:43:59.019473 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:59.019448 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c9985b-f3ab-4e84-8db9-41a954d358a6-kube-api-access-wg9rm" pod "57c9985b-f3ab-4e84-8db9-41a954d358a6" (UID: "57c9985b-f3ab-4e84-8db9-41a954d358a6"). InnerVolumeSpecName "kube-api-access-wg9rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:59.020074 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:59.020050 5300 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57c9985b-f3ab-4e84-8db9-41a954d358a6-maas-api-tls" pod "57c9985b-f3ab-4e84-8db9-41a954d358a6" (UID: "57c9985b-f3ab-4e84-8db9-41a954d358a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:43:59.118381 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:59.118329 5300 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c9985b-f3ab-4e84-8db9-41a954d358a6-openshift-service-ca\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:43:59.118381 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:59.118378 5300 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c9985b-f3ab-4e84-8db9-41a954d358a6-metrics-certs\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:43:59.118381 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:59.118393 5300 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg9rm\" (UniqueName: \"kubernetes.io/projected/57c9985b-f3ab-4e84-8db9-41a954d358a6-kube-api-access-wg9rm\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:43:59.118740 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:59.118407 5300 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c9985b-f3ab-4e84-8db9-41a954d358a6-maas-api-tls\") on node \"ip-10-0-140-206.ec2.internal\" DevicePath \"\"" Oct 30 01:43:59.662794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:59.661803 5300 generic.go:356] "Generic (PLEG): container finished" podID="57c9985b-f3ab-4e84-8db9-41a954d358a6" containerID="e507abfe333c7bc21bbf1009b178bc6b037a51de612e94e8ddf38afcf456b8e7" exitCode=0 Oct 30 01:43:59.662794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:59.661920 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln" event={"ID":"57c9985b-f3ab-4e84-8db9-41a954d358a6","Type":"ContainerDied","Data":"e507abfe333c7bc21bbf1009b178bc6b037a51de612e94e8ddf38afcf456b8e7"} Oct 30 01:43:59.662794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:59.661972 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln" event={"ID":"57c9985b-f3ab-4e84-8db9-41a954d358a6","Type":"ContainerDied","Data":"8868191ba27b5601bd487e322289d347181a6c058779c357c7eafa0f48cd2df1"} Oct 30 01:43:59.662794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:59.662004 5300 scope.go:122] "RemoveContainer" containerID="e507abfe333c7bc21bbf1009b178bc6b037a51de612e94e8ddf38afcf456b8e7" Oct 30 01:43:59.662794 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:59.662166 5300 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln" Oct 30 01:43:59.697611 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:59.697584 5300 scope.go:122] "RemoveContainer" containerID="e507abfe333c7bc21bbf1009b178bc6b037a51de612e94e8ddf38afcf456b8e7" Oct 30 01:43:59.697962 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:43:59.697939 5300 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e507abfe333c7bc21bbf1009b178bc6b037a51de612e94e8ddf38afcf456b8e7\": container with ID starting with e507abfe333c7bc21bbf1009b178bc6b037a51de612e94e8ddf38afcf456b8e7 not found: ID does not exist" containerID="e507abfe333c7bc21bbf1009b178bc6b037a51de612e94e8ddf38afcf456b8e7" Oct 30 01:43:59.698072 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:59.697973 5300 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e507abfe333c7bc21bbf1009b178bc6b037a51de612e94e8ddf38afcf456b8e7"} err="failed to get container status \"e507abfe333c7bc21bbf1009b178bc6b037a51de612e94e8ddf38afcf456b8e7\": rpc error: code = NotFound desc = could not find container \"e507abfe333c7bc21bbf1009b178bc6b037a51de612e94e8ddf38afcf456b8e7\": container with ID starting with e507abfe333c7bc21bbf1009b178bc6b037a51de612e94e8ddf38afcf456b8e7 not found: ID does not exist" Oct 30 01:43:59.755412 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:59.755201 5300 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln"] Oct 30 01:43:59.772134 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:59.772046 5300 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-4k9ln"] Oct 30 01:43:59.821784 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:43:59.819603 5300 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57c9985b-f3ab-4e84-8db9-41a954d358a6" path="/var/lib/kubelet/pods/57c9985b-f3ab-4e84-8db9-41a954d358a6/volumes" Oct 30 01:44:21.590623 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:44:21.590582 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:44:37.294582 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:44:37.294525 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:48.784403 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:44:48.784372 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5dfd774998-7khzv_ef5c2a40-f573-4613-b77f-6b8b72dfc1cc/authorino/0.log" Oct 30 01:44:51.591270 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:44:51.591226 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:45:01.147223 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:01.147140 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84dd548884-rfc58_f5fb34d6-45e8-4caa-95ef-6f801d88a584/copy-manifests/0.log" Oct 30 01:45:01.176111 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:01.176081 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84dd548884-rfc58_f5fb34d6-45e8-4caa-95ef-6f801d88a584/manager/0.log" Oct 30 01:45:01.368671 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:01.368639 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-997pg_917950da-2187-4df2-972b-89c4dc58a547/proxy/0.log" Oct 30 01:45:02.143383 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:02.143341 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-k46sj_bdf8e872-44f9-4c96-9ee4-ff00082048bd/maas-api/0.log" Oct 30 01:45:04.329849 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:04.329819 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-5nb97_4907aff2-e13d-41bc-92c9-6f9a522a81c4/postgres/0.log" Oct 30 01:45:05.965944 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:05.965905 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5dfd774998-7khzv_ef5c2a40-f573-4613-b77f-6b8b72dfc1cc/authorino/0.log" Oct 30 01:45:06.406778 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:06.406740 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-gvl5b_6bae3153-fe09-4948-9f0a-b6815c7f16e1/manager/0.log" Oct 30 01:45:06.587701 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:06.587673 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-rngkx_17297963-0671-42e5-b81c-1dbf7e6ec6bd/kuadrant-console-plugin/0.log" Oct 30 01:45:08.524946 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:08.524916 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6cfd9d8567-zfw8x_c45710be-fadb-4638-a39a-c84664585e3c/kube-auth-proxy/0.log" Oct 30 01:45:09.286662 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:09.286617 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-dzvm2_5f16e1d3-330a-4e81-b059-107bf40670b7/istio-proxy/0.log" Oct 30 01:45:10.246339 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:10.246311 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-pwmfm_66f157e0-a09c-46c7-bd51-43bd5567b9ab/istio-proxy/0.log" Oct 30 01:45:10.424174 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:10.424129 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-scn6w_eea55cea-9b44-45d9-b25f-0b9ca254de04/istio-proxy/0.log" Oct 30 01:45:10.777867 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:10.777836 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-nkqms_7817f490-6fc5-4d17-8acc-f83273bd9a5c/payload-processing/0.log" Oct 30 01:45:12.044893 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:12.044867 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-696nh_ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720/main/0.log" Oct 30 01:45:12.221735 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:12.221702 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-kzbhd_1b27f00f-ab83-4393-9223-8b4e790dfa28/main/0.log" Oct 30 01:45:12.584275 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:12.584237 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-k659t_4dc03dfd-8c9e-42a1-9677-361219162734/main/0.log" Oct 30 01:45:12.771051 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:12.771021 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p_1166fd98-968e-4fda-a0b9-0cfea1ed9e03/main/0.log" Oct 30 01:45:13.649745 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:13.649709 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5dfd774998-7khzv_ef5c2a40-f573-4613-b77f-6b8b72dfc1cc/authorino/0.log" Oct 30 01:45:21.591803 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:45:21.591752 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:45:24.727396 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:24.727363 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84dd548884-rfc58_f5fb34d6-45e8-4caa-95ef-6f801d88a584/copy-manifests/0.log" Oct 30 01:45:24.758326 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:24.758293 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84dd548884-rfc58_f5fb34d6-45e8-4caa-95ef-6f801d88a584/manager/0.log" Oct 30 01:45:24.957790 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:24.957744 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-997pg_917950da-2187-4df2-972b-89c4dc58a547/proxy/0.log" Oct 30 01:45:25.502723 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:25.502696 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-k46sj_bdf8e872-44f9-4c96-9ee4-ff00082048bd/maas-api/0.log" Oct 30 01:45:27.445583 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:27.445555 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-5nb97_4907aff2-e13d-41bc-92c9-6f9a522a81c4/postgres/0.log" Oct 30 01:45:28.505684 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:28.505651 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5dfd774998-7khzv_ef5c2a40-f573-4613-b77f-6b8b72dfc1cc/authorino/0.log" Oct 30 01:45:28.922375 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:28.922292 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-gvl5b_6bae3153-fe09-4948-9f0a-b6815c7f16e1/manager/0.log" Oct 30 01:45:29.109134 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:29.109104 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-rngkx_17297963-0671-42e5-b81c-1dbf7e6ec6bd/kuadrant-console-plugin/0.log" Oct 30 01:45:30.841514 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:30.841485 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6cfd9d8567-zfw8x_c45710be-fadb-4638-a39a-c84664585e3c/kube-auth-proxy/0.log" Oct 30 01:45:31.581546 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:31.581512 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-dzvm2_5f16e1d3-330a-4e81-b059-107bf40670b7/istio-proxy/0.log" Oct 30 01:45:32.509170 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:32.509138 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-pwmfm_66f157e0-a09c-46c7-bd51-43bd5567b9ab/istio-proxy/0.log" Oct 30 01:45:32.678436 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:32.678401 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-scn6w_eea55cea-9b44-45d9-b25f-0b9ca254de04/istio-proxy/0.log" Oct 30 01:45:33.031584 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:33.031558 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-nkqms_7817f490-6fc5-4d17-8acc-f83273bd9a5c/payload-processing/0.log" Oct 30 01:45:34.091405 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:34.091376 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-696nh_ac36d5c9-9a67-4ca1-abc6-f47ff1bcf720/main/0.log" Oct 30 01:45:34.272179 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:34.272149 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-kzbhd_1b27f00f-ab83-4393-9223-8b4e790dfa28/main/0.log" Oct 30 01:45:34.626347 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:34.626301 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-k659t_4dc03dfd-8c9e-42a1-9677-361219162734/main/0.log" Oct 30 01:45:34.802806 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:34.802778 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-jhh6p_1166fd98-968e-4fda-a0b9-0cfea1ed9e03/main/0.log" Oct 30 01:45:37.393043 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:45:37.393012 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:51.591721 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:45:51.591687 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:46:07.798461 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:46:07.798432 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-nkqms_7817f490-6fc5-4d17-8acc-f83273bd9a5c/payload-processing/0.log" Oct 30 01:46:07.988982 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:46:07.988952 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-nkqms_7817f490-6fc5-4d17-8acc-f83273bd9a5c/payload-processing/0.log" Oct 30 01:46:15.457105 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:46:15.457069 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:15.558270 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:46:15.558222 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:21.591945 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:46:21.591902 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:46:37.502822 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:46:37.502716 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:51.123020 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:46:51.122986 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-nkqms_7817f490-6fc5-4d17-8acc-f83273bd9a5c/payload-processing/0.log" Oct 30 01:46:51.325451 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:46:51.325425 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-nkqms_7817f490-6fc5-4d17-8acc-f83273bd9a5c/payload-processing/0.log" Oct 30 01:46:51.591223 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:46:51.591191 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor" Oct 30 01:46:51.668025 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:46:51.667995 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:51.808267 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:46:51.808233 5300 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:56.138643 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:46:56.138611 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xdkdr_b0661ca3-6f0a-4744-9d50-f2baa1a80b45/global-pull-secret-syncer/0.log" Oct 30 01:46:56.637803 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:46:56.637755 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-89gxj_271e26db-6a8b-4e46-afb7-139cbc57c1e1/konnectivity-agent/0.log" Oct 30 01:46:57.136412 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:46:57.136381 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-206.ec2.internal_3b175f0644888f98ec9cdc35355eb1b8/haproxy/0.log" Oct 30 01:47:03.760517 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:03.760486 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5dfd774998-7khzv_ef5c2a40-f573-4613-b77f-6b8b72dfc1cc/authorino/0.log" Oct 30 01:47:03.959523 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:03.959500 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-gvl5b_6bae3153-fe09-4948-9f0a-b6815c7f16e1/manager/0.log" Oct 30 01:47:04.079987 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:04.079919 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-rngkx_17297963-0671-42e5-b81c-1dbf7e6ec6bd/kuadrant-console-plugin/0.log" Oct 30 01:47:05.578640 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:05.578608 5300 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tvzh4/perf-node-gather-daemonset-s6wdt"] Oct 30 01:47:05.579036 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:05.578982 5300 cpu_manager.go:414] "containerMap: removing container" podUID="57c9985b-f3ab-4e84-8db9-41a954d358a6" containerName="maas-api" Oct 30 01:47:05.579036 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:05.578993 5300 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57c9985b-f3ab-4e84-8db9-41a954d358a6" containerName="maas-api" Oct 30 01:47:05.579114 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:05.579055 5300 memory_manager.go:365] "RemoveStaleState removing state" podUID="57c9985b-f3ab-4e84-8db9-41a954d358a6" containerName="maas-api" Oct 30 01:47:05.582176 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:05.582160 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-s6wdt" Oct 30 01:47:05.601211 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:05.601186 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tvzh4/perf-node-gather-daemonset-s6wdt"] Oct 30 01:47:05.650522 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:05.650499 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5da3b34e-0adc-4a80-aca8-be8330de79f0-lib-modules\") pod \"perf-node-gather-daemonset-s6wdt\" (UID: \"5da3b34e-0adc-4a80-aca8-be8330de79f0\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-s6wdt" Oct 30 01:47:05.650666 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:05.650532 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5da3b34e-0adc-4a80-aca8-be8330de79f0-proc\") pod \"perf-node-gather-daemonset-s6wdt\" (UID: \"5da3b34e-0adc-4a80-aca8-be8330de79f0\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-s6wdt" Oct 30 01:47:05.650666 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:05.650561 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5da3b34e-0adc-4a80-aca8-be8330de79f0-podres\") pod \"perf-node-gather-daemonset-s6wdt\" (UID: \"5da3b34e-0adc-4a80-aca8-be8330de79f0\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-s6wdt" Oct 30 01:47:05.650748 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:05.650676 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5da3b34e-0adc-4a80-aca8-be8330de79f0-sys\") pod \"perf-node-gather-daemonset-s6wdt\" (UID: \"5da3b34e-0adc-4a80-aca8-be8330de79f0\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-s6wdt" Oct 30 01:47:05.650748 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:05.650724 5300 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t7w9\" (UniqueName: \"kubernetes.io/projected/5da3b34e-0adc-4a80-aca8-be8330de79f0-kube-api-access-9t7w9\") pod \"perf-node-gather-daemonset-s6wdt\" (UID: \"5da3b34e-0adc-4a80-aca8-be8330de79f0\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-s6wdt" Oct 30 01:47:05.683534 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:05.683516 5300 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tvzh4\"/\"default-dockercfg-lc9qq\"" Oct 30 01:47:05.689405 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:05.689390 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tvzh4\"/\"kube-root-ca.crt\"" Oct 30 01:47:05.689518 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:05.689502 5300 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tvzh4\"/\"openshift-service-ca.crt\"" Oct 30 01:47:05.751639 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:05.751614 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5da3b34e-0adc-4a80-aca8-be8330de79f0-podres\") pod \"perf-node-gather-daemonset-s6wdt\" (UID: \"5da3b34e-0adc-4a80-aca8-be8330de79f0\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-s6wdt" Oct 30 01:47:05.751787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:05.751665 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5da3b34e-0adc-4a80-aca8-be8330de79f0-sys\") pod \"perf-node-gather-daemonset-s6wdt\" (UID: \"5da3b34e-0adc-4a80-aca8-be8330de79f0\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-s6wdt" Oct 30 01:47:05.751787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:05.751699 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t7w9\" (UniqueName: \"kubernetes.io/projected/5da3b34e-0adc-4a80-aca8-be8330de79f0-kube-api-access-9t7w9\") pod \"perf-node-gather-daemonset-s6wdt\" (UID: \"5da3b34e-0adc-4a80-aca8-be8330de79f0\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-s6wdt" Oct 30 01:47:05.751787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:05.751731 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5da3b34e-0adc-4a80-aca8-be8330de79f0-lib-modules\") pod \"perf-node-gather-daemonset-s6wdt\" (UID: \"5da3b34e-0adc-4a80-aca8-be8330de79f0\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-s6wdt" Oct 30 01:47:05.751787 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:05.751747 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5da3b34e-0adc-4a80-aca8-be8330de79f0-podres\") pod \"perf-node-gather-daemonset-s6wdt\" (UID: \"5da3b34e-0adc-4a80-aca8-be8330de79f0\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-s6wdt" Oct 30 01:47:05.751952 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:05.751756 5300 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5da3b34e-0adc-4a80-aca8-be8330de79f0-proc\") pod \"perf-node-gather-daemonset-s6wdt\" (UID: \"5da3b34e-0adc-4a80-aca8-be8330de79f0\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-s6wdt" Oct 30 01:47:05.751952 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:05.751815 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5da3b34e-0adc-4a80-aca8-be8330de79f0-sys\") pod \"perf-node-gather-daemonset-s6wdt\" (UID: \"5da3b34e-0adc-4a80-aca8-be8330de79f0\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-s6wdt" Oct 30 01:47:05.751952 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:05.751853 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5da3b34e-0adc-4a80-aca8-be8330de79f0-lib-modules\") pod \"perf-node-gather-daemonset-s6wdt\" (UID: \"5da3b34e-0adc-4a80-aca8-be8330de79f0\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-s6wdt" Oct 30 01:47:05.751952 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:05.751865 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5da3b34e-0adc-4a80-aca8-be8330de79f0-proc\") pod \"perf-node-gather-daemonset-s6wdt\" (UID: \"5da3b34e-0adc-4a80-aca8-be8330de79f0\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-s6wdt" Oct 30 01:47:05.773479 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:05.773453 5300 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t7w9\" (UniqueName: \"kubernetes.io/projected/5da3b34e-0adc-4a80-aca8-be8330de79f0-kube-api-access-9t7w9\") pod \"perf-node-gather-daemonset-s6wdt\" (UID: \"5da3b34e-0adc-4a80-aca8-be8330de79f0\") " pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-s6wdt" Oct 30 01:47:05.893562 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:05.893486 5300 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-s6wdt" Oct 30 01:47:06.190336 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:06.190306 5300 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tvzh4/perf-node-gather-daemonset-s6wdt"] Oct 30 01:47:06.192198 ip-10-0-140-206 kubenswrapper[5300]: W1030 01:47:06.192165 5300 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5da3b34e_0adc_4a80_aca8_be8330de79f0.slice/crio-2a55262f0caff8b86dfbb8e30ef1ffda34fc0b273360180074ad0487ba4ef1c8 WatchSource:0}: Error finding container 2a55262f0caff8b86dfbb8e30ef1ffda34fc0b273360180074ad0487ba4ef1c8: Status 404 returned error can't find the container with id 2a55262f0caff8b86dfbb8e30ef1ffda34fc0b273360180074ad0487ba4ef1c8 Oct 30 01:47:06.523169 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:06.523132 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-s6wdt" event={"ID":"5da3b34e-0adc-4a80-aca8-be8330de79f0","Type":"ContainerStarted","Data":"0dcea71028ca1ef823d60427206fa3a104047f91acbc6d61c620284822968b1e"} Oct 30 01:47:06.523375 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:06.523189 5300 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-s6wdt" event={"ID":"5da3b34e-0adc-4a80-aca8-be8330de79f0","Type":"ContainerStarted","Data":"2a55262f0caff8b86dfbb8e30ef1ffda34fc0b273360180074ad0487ba4ef1c8"} Oct 30 01:47:06.523375 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:06.523304 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-s6wdt" Oct 30 01:47:06.565796 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:06.565724 5300 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-s6wdt" podStartSLOduration=1.565709306 podStartE2EDuration="1.565709306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:06.564789482 +0000 UTC m=+3826.880657823" watchObservedRunningTime="2026-10-30 01:47:06.565709306 +0000 UTC m=+3826.881577620" Oct 30 01:47:07.165204 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:07.165172 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-vmstj_67bde315-b2ea-4d17-88d8-f290cd2c994e/monitoring-plugin/0.log" Oct 30 01:47:07.942471 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:07.942441 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vngsd_fcf5056b-ef4e-430b-8b5c-83fad65af440/node-exporter/0.log" Oct 30 01:47:08.053094 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:08.053064 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vngsd_fcf5056b-ef4e-430b-8b5c-83fad65af440/kube-rbac-proxy/0.log" Oct 30 01:47:08.162656 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:08.162630 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vngsd_fcf5056b-ef4e-430b-8b5c-83fad65af440/init-textfile/0.log" Oct 30 01:47:08.600610 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:08.600581 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b85f5ff6-7882-40e1-a28a-1cb8cf13c5be/prometheus/0.log" Oct 30 01:47:08.710008 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:08.709985 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b85f5ff6-7882-40e1-a28a-1cb8cf13c5be/config-reloader/0.log" Oct 30 01:47:08.819297 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:08.819276 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b85f5ff6-7882-40e1-a28a-1cb8cf13c5be/thanos-sidecar/0.log" Oct 30 01:47:08.928169 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:08.928103 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b85f5ff6-7882-40e1-a28a-1cb8cf13c5be/kube-rbac-proxy-web/0.log" Oct 30 01:47:09.038127 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:09.038097 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b85f5ff6-7882-40e1-a28a-1cb8cf13c5be/kube-rbac-proxy/0.log" Oct 30 01:47:09.148675 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:09.148651 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b85f5ff6-7882-40e1-a28a-1cb8cf13c5be/kube-rbac-proxy-thanos/0.log" Oct 30 01:47:09.263108 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:09.263079 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b85f5ff6-7882-40e1-a28a-1cb8cf13c5be/init-config-reloader/0.log" Oct 30 01:47:09.383639 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:09.383607 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-g9xp7_3c78afce-0bbb-49e1-aceb-ae4a69844075/prometheus-operator/0.log" Oct 30 01:47:09.487469 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:09.487441 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-g9xp7_3c78afce-0bbb-49e1-aceb-ae4a69844075/kube-rbac-proxy/0.log" Oct 30 01:47:12.537632 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:12.537606 5300 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tvzh4/perf-node-gather-daemonset-s6wdt" Oct 30 01:47:13.821975 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:13.821946 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-698568d77b-q4qgw_09c85662-2f01-4d2d-9a04-24e3245549f3/console/0.log" Oct 30 01:47:17.211690 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:17.211653 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5jhd7_81258c01-d653-4a05-87ca-8373e7fad83d/dns/0.log" Oct 30 01:47:17.322011 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:17.321981 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5jhd7_81258c01-d653-4a05-87ca-8373e7fad83d/kube-rbac-proxy/0.log" Oct 30 01:47:17.875637 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:17.875609 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6mjml_995884d0-cb39-4c07-9dbe-5cd3529d5908/dns-node-resolver/0.log" Oct 30 01:47:19.274587 ip-10-0-140-206 kubenswrapper[5300]: I1030 01:47:19.274552 5300 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g9ftg_5dff88d3-2e35-40e9-8aea-208519485adc/node-ca/0.log" Oct 30 01:47:21.592153 ip-10-0-140-206 kubenswrapper[5300]: E1030 01:47:21.592120 5300 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xmsbl" containerName="extractor"